Nov 01 03:39:08.583309 ip-10-0-129-252 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 01 03:39:08.583322 ip-10-0-129-252 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 01 03:39:08.583331 ip-10-0-129-252 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 01 03:39:08.583727 ip-10-0-129-252 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 01 03:39:18.832933 ip-10-0-129-252 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 01 03:39:18.832948 ip-10-0-129-252 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d37aa4d809344a978c9c2092319a0c49 --
Nov 01 03:41:27.204712 ip-10-0-129-252 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 03:41:27.598189 ip-10-0-129-252 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:41:27.598189 ip-10-0-129-252 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 03:41:27.598189 ip-10-0-129-252 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:41:27.598189 ip-10-0-129-252 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:41:27.602521 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602443    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:41:27.602521 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602520    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:41:27.602521 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602524    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602528    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602538    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602543    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602547    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602550    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602553    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602555    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602558    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602561    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602563    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602566    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602569    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602572    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602574    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602583    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602586    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602589    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602591    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:41:27.602611 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602594    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602596    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602599    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602601    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602604    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602607    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602610    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602613    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602615    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602618    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602621    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602624    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602627    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602629    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602632    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602634    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602637    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602640    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602642    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:41:27.603053 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602645    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602648    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602652    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602656    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602659    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602662    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602665    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602668    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602671    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602673    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602676    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602678    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602681    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602684    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602686    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602690    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602693    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602695    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602698    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602701    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:41:27.603510 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602704    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602707    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602711    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602714    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602716    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602719    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602721    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602724    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602726    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602729    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602732    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602735    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602737    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602740    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602742    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602745    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602747    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602749    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602752    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:41:27.603972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602755    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602758    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602761    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602763    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602766    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602768    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602771    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602773    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602776    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602779    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602781    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602784    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602787    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602789    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602792    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602794    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602797    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602799    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602802    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602804    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:41:27.604452 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602807    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602809    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602812    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602815    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602817    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602819    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.602822    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603624    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603630    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603635    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603638    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603641    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603644    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603646    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603650    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603652    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603655    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603658    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603660    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:41:27.604950 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603664    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603667    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603670    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603673    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603675    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603678    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603680    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603683    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603686    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603688    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603691    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603695    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603698    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603702    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603705    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603708    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603710    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603713    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603715    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:41:27.605403 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603718    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603720    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603723    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603725    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603728    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603731    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603733    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603735    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603738    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603741    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603744    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603746    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603749    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603752    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603755    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603758    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603761    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603763    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603766    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603768    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:41:27.605966 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603771    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603773    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603776    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603778    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603781    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603783    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603786    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603788    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603791    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603793    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603796    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603799    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603801    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603804    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603806    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603809    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603811    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603814    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603816    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:41:27.606449 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603819    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603822    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603825    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603828    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603831    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603833    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603836    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603838    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603841    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603843    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603846    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603848    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603851    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603853    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603856    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603859    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603861    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603864    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603866    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603869    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:41:27.606921 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603871    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603874    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603876    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603879    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603882    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603884    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603887    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603890    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603892    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603895    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603897    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603901    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603905    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603909    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603912    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.603915    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604443    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604449    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604452    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604455    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:41:27.607391 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604459    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604461    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604464    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604467    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604470    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604473    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604475    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604478    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604481    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604483    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604486    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604488    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604494    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604497    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604501    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604506    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604509    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604512    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604515    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604517    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:41:27.607881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604520    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604523    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604525    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604528    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604530    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604533    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604536    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604539    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604542    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604545    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604548    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604550    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604553    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604555    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604558    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604561    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604564    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604566    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604569    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604571    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:41:27.608341 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604574    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604576    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604579    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604581    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604584    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604587    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604589    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604592    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604595    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604597    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604600    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604602    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604605    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604607    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604610    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604612    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604615    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604618    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:41:27.608833 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604621    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604623    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604626    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604629    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604632    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604634    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604637    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604639    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604642    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604644    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604647    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604649    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604652    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604654    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604656    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604659    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604663    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604666    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604669    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:41:27.609249 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604672    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604675    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604677    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604680    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604683    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604685    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604696    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604699    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604701    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604704    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604706    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604709    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604712    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604714    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604717    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604720    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604724    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604727    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604730    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604733    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:41:27.609722 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604735    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604738    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604740    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604743    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.604746    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604841    2583 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604849    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604858    2583 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604865    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604871    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604876    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604881    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604885    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604888    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604891    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604895    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604898    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604901    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604904    2583 flags.go:64] FLAG: --cgroup-root=""
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604907    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604910    2583 flags.go:64] FLAG: --client-ca-file=""
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604913    2583 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604916    2583 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 03:41:27.610182 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604919    2583 flags.go:64] FLAG: --cluster-domain=""
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604922    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604925    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604929    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604932    2583 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604936    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604939    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604943    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604946    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604949    2583 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604953    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604956    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604959    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604962    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604968    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604971    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604975    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604977    2583 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604980    2583 flags.go:64] FLAG: --enable-server="true"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604983    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604988    2583 flags.go:64] FLAG: --event-burst="100"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604991    2583 flags.go:64] FLAG: --event-qps="50"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604994    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.604997    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 03:41:27.610716 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605000    2583 flags.go:64] FLAG: --eviction-hard=""
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605004    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605007    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605010    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605013    2583 flags.go:64] FLAG: --eviction-soft=""
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605016    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605019    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605022    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605025    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605028    2583 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605031    2583 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605034    2583 flags.go:64] FLAG: --feature-gates=""
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605038    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605041    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605044    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605047    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605050    2583 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605054    2583 flags.go:64] FLAG: --help="false"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605056    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605060    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605063    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605065    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605071    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 03:41:27.611286 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605074    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605077    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605080    2583 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605083    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605085    2583 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605088    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605091    2583 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605094    2583 flags.go:64] FLAG: --kube-reserved=""
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605100    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605103    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605107    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605109    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605112    2583 flags.go:64] FLAG: --lock-file=""
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605115    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605118    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605121    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605127    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605130    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605133    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605136    2583 flags.go:64] FLAG: --logging-format="text"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605139    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605142    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605145    2583 flags.go:64] FLAG: --manifest-url=""
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605148    2583 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605152    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 03:41:27.611839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605155    2583 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605159    2583 flags.go:64] FLAG: --max-pods="110"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605162    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605166    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605168    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605171    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605176    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605179    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605182    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605188    2583 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605191    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605194    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605197    2583 flags.go:64] FLAG: --pod-cidr=""
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605200    2583 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605203    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605208    2583 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605211    2583 flags.go:64] FLAG: --port="10250"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605214    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605217    2583 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01677bd4104f0690c"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605220    2583 flags.go:64] FLAG: --qos-reserved=""
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605223    2583 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605226    2583 flags.go:64] FLAG: --register-node="true"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605229    2583 flags.go:64] FLAG: --register-with-taints=""
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605233    2583 flags.go:64] FLAG: --registry-burst="10"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605236    2583 flags.go:64] FLAG: --registry-qps="5"
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605239    2583 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 03:41:27.612419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605242    2583 flags.go:64] FLAG: --reserved-memory=""
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605246    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605249    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605252    2583 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605255    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605258    2583 flags.go:64] FLAG: --runonce="false"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605261    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605264    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605267    2583 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605270    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605273    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605276    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605281    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605284    2583 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605287    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605290    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605293    2583 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605296    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605299    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605302    2583 flags.go:64] FLAG: --system-cgroups=""
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605305    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605311    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605314    2583 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605317    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605321    2583 flags.go:64] FLAG: --tls-min-version=""
Nov 01 03:41:27.613060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605324    2583 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605327    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605330    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605333    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605336    2583 flags.go:64] FLAG: --v="2"
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605340    2583 flags.go:64] FLAG: --version="false"
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605344    2583 flags.go:64] FLAG: --vmodule=""
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605348    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605351    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605492    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605497    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605500    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605503    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605506    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605508    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605511    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605514    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605517    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605520    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605523    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605526    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:41:27.613665 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605529    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605531    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605534    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605537    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605540    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605542    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605545    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605548    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605551    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605554    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605557    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605559    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605562    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605564    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605567    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605570    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605572    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605575    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605577    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605580    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:41:27.614157 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605582    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605585    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605588    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605590    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605592    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605595    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605598    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605600    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605603    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605606    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605610    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605612    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605615    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605618    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605620    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605622    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605625    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605627    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605630    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605634    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:41:27.614638 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605636    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605639    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605641    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605644    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605647    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605649    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605652    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605654    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605657    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605660    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605662    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605665    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605668    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605670    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605673    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605675    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605678    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605680    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605683    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605685    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:41:27.615116 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605688    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605690    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605695    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605697    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605700    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605703    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605705    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605708    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605711    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605713    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605716    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605720    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605722    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605725    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605727    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605730    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605733    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605735    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605738    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:41:27.615697 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605740    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605743    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605746    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605750    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605753    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605756    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605759    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605763    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605766    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605770    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605774    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605777    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605780    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605782    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605785    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:41:27.616139 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.605844    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605889    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605893    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605896    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605899    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605902    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605905    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605908    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605911    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605913    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605917    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605920    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605922    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605925    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605928    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605931    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605933    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605936    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605938    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:41:27.616512 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605941    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605943    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605946    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605948    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605951    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605953    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605956    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605958    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605961    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605964    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605967    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605969    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605972    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605977    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605980    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605983    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605986    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605988    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605991    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:41:27.616964 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605993    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605996    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.605999    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606003    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606008    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606011    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606014    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606016    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606019    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606023    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606027    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606029    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606033    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606035    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606038    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606042    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606045    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606048    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606050    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:41:27.617393 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606053    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606056    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606058    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606061    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606063    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606066    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606069    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606072    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606074    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606077    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606080    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606083    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606085    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606088    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606090    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606092    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606095    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606099    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606102    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:41:27.617845 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606104    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606107    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606109    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606112    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606114    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606117    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606120    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606122    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606125    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606128    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606130    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606133    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606135    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606138    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606141    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606144    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606146    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606149    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606151    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606154    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:41:27.618289 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606156    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606159    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606161    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606169    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606172    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606174    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606177    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606179    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606182    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606184    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.606188    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.614920    2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.614937    2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.614951    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.614955    2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615017    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615022    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615026    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615029    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:41:27.618785 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615032    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615034    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615037    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615040    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615043    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615045    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615048    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615050    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615053    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615056    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615058    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615061    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615064    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615066    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615069    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615071    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615074    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615076    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615079    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615081    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:41:27.619227 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615084    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615086    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615089    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615091    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615094    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615106    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615109    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615113    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615119    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615123    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615125    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615128    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615130    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615133    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615136    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615139    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615143    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615146    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615149    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:41:27.619762 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615152    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615154    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615157    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615159    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615162    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615165    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615167    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615170    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615172    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615175    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615177    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615180    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615182    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615185    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615187    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615190    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615192    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615195    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615198    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615206    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:41:27.620207 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615209    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615212    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615214    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615217    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615219    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615222    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615224    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615227    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615229    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615232    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615234    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615237    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615239    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615242    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615244    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615247    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615249    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615252    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615254    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615257    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:41:27.620677 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615259    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615262    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615265    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615267    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615270    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615272    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615275    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615278    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615280    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615283    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615286    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615289    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615297    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615300    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615303    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615305    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615308    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615310    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615313    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615316    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:41:27.621191 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615318    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615321    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615323    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.615351    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615559    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615564    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615567    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615570    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615573    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615576    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615578    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615581    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615584    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615586    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615589    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615591    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:41:27.621671 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615594    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615597    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615599    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615602    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615604    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615607    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615610    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615613    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615616    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615624    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615627    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615629    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615632    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615634    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615637    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615639    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615642    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615644    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615647    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:41:27.622054 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615650    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615652    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615654    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615657    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615659    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615662    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615664    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615667    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615669    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615672    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615674    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615677    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615679    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615682    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615684    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615687    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615689    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615692    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615695    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615697    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:41:27.622514 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615700    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615703    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615705    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615715    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615718    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615720    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615723    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615725    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615728    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615730    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615732    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615735    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615737    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615740    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615742    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615745    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615747    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615750    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615752    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:41:27.622959 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615755    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615757    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615760    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615762    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615764    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615773    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615775    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615778    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615780    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615783    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615786    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615788    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615791    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615793    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615795    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615798    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615800    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615808    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615811    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:41:27.623404 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615813    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615817    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615821    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615824    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615828    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615830    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615833    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615835    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615838    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615841    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615843    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615847    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615851    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615854    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615857    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615859    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:41:27.623925 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615862    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.615886    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615923    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615926    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615929    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615932    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615934    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615937    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615940    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615942    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615944    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615947    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615949    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615952    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615954    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615963    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:41:27.624295 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615965    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615968    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615972    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615976    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615978    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615981    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615983    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615986    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615989    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615991    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615994    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.615996    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616000    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616004    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616007    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616009    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616012    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616014    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616017    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:41:27.624739 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616020    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616022    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616024    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616027    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616030    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616032    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616035    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616037    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616040    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616042    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616045    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616047    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616049    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616057    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616060    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616063    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616065    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616067    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616070    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616073    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:41:27.625172 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616076    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616078    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616081    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616083    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616086    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616088    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616091    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616093    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616096    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616098    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616100    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616103    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616106    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616108    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616111    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616113    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616116    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616118    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616121    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:41:27.625772 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616123    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616126    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616128    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616131    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616133    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616136    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616138    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616146    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616149    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616152    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616154    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616157    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616159    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616162    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616167    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616170    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616172    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616175    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616177    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616180    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:41:27.626220 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616182    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616185    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616188    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616190    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616193    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616196    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616199    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616201    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616204    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616206    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616209    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616211    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616213    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:27.616216    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.616269    2583 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.618689    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.619819    2583 server.go:1014] "Starting client certificate rotation"
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.619915    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 03:41:27.626700 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.620729    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 03:41:27.627137 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.622274    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:41:27.627137 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.623609    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:41:27.634567 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.634549    2583 log.go:25] "Validated CRI v1 runtime API"
Nov 01 03:41:27.638522 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.638510    2583 log.go:25] "Validated CRI v1 image API"
Nov 01 03:41:27.639273 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.639252    2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 03:41:27.644118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.644097    2583 fs.go:135] Filesystem UUIDs: map[4539a4fa-ff2f-412d-aa88-0239caed6907:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7b0db276-8dcb-45d7-8dbc-510c220568f7:/dev/nvme0n1p3]
Nov 01 03:41:27.644178 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.644119    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 03:41:27.649197 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.649087    2583 manager.go:217] Machine: {Timestamp:2026-11-01 03:41:27.647624552 +0000 UTC m=+0.350856392 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101915 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e7816fc5c295025e4fd9cfea46659 SystemUUID:ec2e7816-fc5c-2950-25e4-fd9cfea46659 BootID:d37aa4d8-0934-4a97-8c9c-2092319a0c49 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:dc:13:c0:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:dc:13:c0:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:a0:16:c2:4c:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 03:41:27.649197 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.649196    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 03:41:27.649319 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.649299    2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 03:41:27.652714 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.652685    2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 03:41:27.652844 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.652715    2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-252.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 03:41:27.652891 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.652846    2583 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 03:41:27.652891 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.652854    2583 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 03:41:27.652891 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.652887    2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:41:27.652990 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.652903    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:41:27.654155 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.654145    2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 03:41:27.658854 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.658843    2583 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 03:41:27.658964 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.658955    2583 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 03:41:27.659802 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.659791    2583 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 03:41:27.659861 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.659815    2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 03:41:27.659861 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.659829    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 03:41:27.659861 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.659838    2583 kubelet.go:405] "Adding apiserver pod source"
Nov 01 03:41:27.659861 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.659857    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 03:41:27.660154 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.660139    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:41:27.662630 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.662615    2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 03:41:27.669808 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.669793    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:41:27.670754 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.670739    2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 03:41:27.670842 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.670779    2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 03:41:27.670992 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.670978    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 03:41:27.671060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.671006    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 03:41:27.671060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.671021    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 03:41:27.671060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.671032    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 03:41:27.671060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.671041    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 03:41:27.671060 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.671050    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 03:41:27.671682 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.671671    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 03:41:27.671733 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.671686    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 03:41:27.671733 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.671700    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 03:41:27.671733 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.671711    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 03:41:27.671733 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.671732    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 03:41:27.671941 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.671746    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 03:41:27.672497 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.672485    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 03:41:27.672547 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.672501    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 03:41:27.676228 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.676213    2583 server.go:1263] "Started kubelet"
Nov 01 03:41:27.676344 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.676321    2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 03:41:27.676390 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.676327    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 03:41:27.676446 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.676413    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 03:41:27.676935 ip-10-0-129-252 systemd[1]: Started Kubernetes Kubelet.
Nov 01 03:41:27.678259 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.678094    2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 03:41:27.678841 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.678831    2583 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 03:41:27.682984 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.682956    2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 03:41:27.686388 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.686366    2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 03:41:27.686388 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.686381    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 03:41:27.686583 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.686538    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:27.687188 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687168    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 03:41:27.687188 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687168    2583 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 03:41:27.687331 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687200    2583 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 03:41:27.687331 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687277    2583 factory.go:55] Registering systemd factory
Nov 01 03:41:27.687331 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687292    2583 factory.go:223] Registration of the systemd container factory successfully
Nov 01 03:41:27.687331 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687330    2583 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 03:41:27.687513 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687341    2583 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 03:41:27.687579 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687564    2583 factory.go:153] Registering CRI-O factory
Nov 01 03:41:27.687579 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687581    2583 factory.go:223] Registration of the crio container factory successfully
Nov 01 03:41:27.687713 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687629    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 03:41:27.687713 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687647    2583 factory.go:103] Registering Raw factory
Nov 01 03:41:27.687713 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.687659    2583 manager.go:1199] Started watching for new ooms in manager
Nov 01 03:41:27.688727 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.688713    2583 manager.go:319] Starting recovery of all containers
Nov 01 03:41:27.701032 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.701019    2583 manager.go:324] Recovery completed
Nov 01 03:41:27.706656 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.706643    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:41:27.708467 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.708449    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:41:27.708533 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.708486    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:41:27.708533 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.708502    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:41:27.709063 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.709046    2583 cpu_manager.go:226] "Starting" policy="none"
Nov 01 03:41:27.709063 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.709060    2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 03:41:27.709180 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.709081    2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 03:41:27.709180 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.709081    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:41:27.709746 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.709732    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:41:27.709806 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.709756    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:41:27.709806 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.709765    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:41:27.711309 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.711293    2583 policy_none.go:50] "Start"
Nov 01 03:41:27.711369 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.711312    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 03:41:27.711369 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.711325    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 03:41:27.713414 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.713401    2583 policy_none.go:44] "Start"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.745012    2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.746162    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.747320    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.747334    2583 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.747350    2583 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.747380    2583 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.747519    2583 manager.go:341] "Starting Device Plugin manager"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.747920    2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.747932    2583 server.go:87] "Starting device plugin registration server"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.748158    2583 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.748170    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.748561    2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.748630    2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.748638    2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.750494    2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 03:41:27.759702 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.750543    2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:27.812642 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.812608    2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-252.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 03:41:27.813810 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.812609    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-252.us-west-2.compute.internal.18e3ced443020ce6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-252.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-252.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 03:41:27.67618583 +0000 UTC m=+0.379417671,LastTimestamp:2026-11-01 03:41:27.67618583 +0000 UTC m=+0.379417671,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-252.us-west-2.compute.internal,}"
Nov 01 03:41:27.848391 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.848339    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:41:27.848955 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.848921    2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"]
Nov 01 03:41:27.849020 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.848995    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:41:27.849133 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.849117    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:41:27.849193 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.849152    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:41:27.849193 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.849169    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:41:27.849281 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.849199    2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.849714 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.849698    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:41:27.849770 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.849725    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:41:27.849770 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.849735    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:41:27.851857 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.851845    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:41:27.851992 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.851978    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.852037 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.852010    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:41:27.852559 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.852541    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:41:27.852633 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.852570    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:41:27.852633 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.852573    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:41:27.852633 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.852594    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:41:27.852633 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.852604    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:41:27.852633 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.852579    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:41:27.854741 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.854725    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.854816 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.854750    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:41:27.855441 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.855417    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:41:27.855505 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.855448    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:41:27.855505 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.855457    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:41:27.857298 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.857284    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-252.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 03:41:27.863699 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.863675    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 01 03:41:27.863699 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.863676    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-252.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 01 03:41:27.863699 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.863680    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 01 03:41:27.863894 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.863809    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 01 03:41:27.863973 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.863896    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-252.us-west-2.compute.internal.18e3ced444ee9cf2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-252.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-252.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-252.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 03:41:27.708466418 +0000 UTC m=+0.411698263,LastTimestamp:2026-11-01 03:41:27.708466418 +0000 UTC m=+0.411698263,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-252.us-west-2.compute.internal,}"
Nov 01 03:41:27.884974 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.884957    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found" node="ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.887548 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.887530    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found" node="ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.888602 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.888586    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/349f3d418ab214cf804b3c489f47ad32-config\") pod \"kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"349f3d418ab214cf804b3c489f47ad32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.888674 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.888609    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.888674 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.888626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.908969 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.908948    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 03:41:27.909023 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.908999    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gzf59"
Nov 01 03:41:27.984805 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.984779    2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.984805 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:27.984805    2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-252.us-west-2.compute.internal\": node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:27.989351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.989335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.989397 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.989359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/349f3d418ab214cf804b3c489f47ad32-config\") pod \"kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"349f3d418ab214cf804b3c489f47ad32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.989397 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.989374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.989513 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.989445    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.989513 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.989448    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:27.989513 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:27.989479    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/349f3d418ab214cf804b3c489f47ad32-config\") pod \"kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"349f3d418ab214cf804b3c489f47ad32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:28.101738 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:28.101685    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:28.189012 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:28.188993    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:28.194093 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:28.194078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:28.202155 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:28.202139    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:28.302666 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:28.302519    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:28.403405 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:28.403350    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:28.504551 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:28.504526    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:28.604817 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:28.604790    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:28.619535 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:28.619511    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 03:41:28.619729 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:28.619648    2583 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://afdb05243526b4c0e91ef2f637640017-e887bece4d7f5b9e.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-252.us-west-2.compute.internal.18e3ced444ef0661\": read tcp 10.0.129.252:33694->13.184.31.129:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-252.us-west-2.compute.internal.18e3ced444ef0661  default   4852 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-252.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-252.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-252.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 03:41:27 +0000 UTC,LastTimestamp:2026-11-01 03:41:27.849730482 +0000 UTC m=+0.552962322,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-252.us-west-2.compute.internal,}"
Nov 01 03:41:28.705583 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:28.705525    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:28.806531 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:28.806480    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:28.907655 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:28.907613    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:28.915370 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:28.915341    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 03:36:28 +0000 UTC" deadline="2028-04-27 01:42:30.811129049 +0000 UTC"
Nov 01 03:41:28.915370 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:28.915367    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13030h1m1.895764839s"
Nov 01 03:41:28.951828 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:28.951812    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:41:28.952769 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:28.952753    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:41:28.952840 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:28.952790    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:41:28.952840 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:28.952807    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:41:29.008635 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:29.008580    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:29.072773 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.072747    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:41:29.108818 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:29.108793    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:29.154137 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.154107    2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:41:29.154137 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.154109    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:41:29.209216 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:29.209186    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:29.310087 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:29.310002    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:29.411148 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:29.411120    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:41:29.482342 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.482311    2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:41:29.486585 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.486563    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:29.557710 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.557676    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:41:29.558720 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.558702    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:41:29.623972 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.623950    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:41:29.635492 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:29.635457    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349f3d418ab214cf804b3c489f47ad32.slice/crio-a3f7f09bb225b48f6c9a5c9f2adc869de435152a098c04e5c0a484be56f891e5 WatchSource:0}: Error finding container a3f7f09bb225b48f6c9a5c9f2adc869de435152a098c04e5c0a484be56f891e5: Status 404 returned error can't find the container with id a3f7f09bb225b48f6c9a5c9f2adc869de435152a098c04e5c0a484be56f891e5
Nov 01 03:41:29.637338 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:29.637317    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d0b123ba86397e4d3cfebb9bec2811.slice/crio-c43b6b72bd59316f0febc6194f4dcdd821474d757ab5392f96bd9a800c60abe6 WatchSource:0}: Error finding container c43b6b72bd59316f0febc6194f4dcdd821474d757ab5392f96bd9a800c60abe6: Status 404 returned error can't find the container with id c43b6b72bd59316f0febc6194f4dcdd821474d757ab5392f96bd9a800c60abe6
Nov 01 03:41:29.643616 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.643601    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:41:29.662568 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.662548    2583 apiserver.go:52] "Watching apiserver"
Nov 01 03:41:29.688606 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.688579    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 03:41:29.754579 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.754531    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal" event={"ID":"349f3d418ab214cf804b3c489f47ad32","Type":"ContainerStarted","Data":"a3f7f09bb225b48f6c9a5c9f2adc869de435152a098c04e5c0a484be56f891e5"}
Nov 01 03:41:29.755638 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.755615    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal" event={"ID":"a5d0b123ba86397e4d3cfebb9bec2811","Type":"ContainerStarted","Data":"c43b6b72bd59316f0febc6194f4dcdd821474d757ab5392f96bd9a800c60abe6"}
Nov 01 03:41:29.779948 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.778779    2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 03:41:29.779948 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.779492    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv","openshift-dns/node-resolver-t8tdn","openshift-network-diagnostics/network-check-target-hbgqw","openshift-image-registry/node-ca-fwzpl","openshift-multus/multus-m7dq2","kube-system/global-pull-secret-syncer-zdw9t","kube-system/konnectivity-agent-mm8h5","openshift-multus/multus-additional-cni-plugins-jz78s","openshift-multus/network-metrics-daemon-vqrc7","openshift-cluster-node-tuning-operator/tuned-kwns5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-g9vql","openshift-ovn-kubernetes/ovnkube-node-5tws4"]
Nov 01 03:41:29.783608 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.783579    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:41:29.785934 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.785912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.788144 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.788127    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.790463 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.790428    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:29.790558 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:29.790517    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:29.792846 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.792830    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:29.792902 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:29.792882    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:29.795818 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.795799    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.798316 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.798302    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:29.798401 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:29.798359    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:29.798749 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.798728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-conf\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.798840 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.798754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-systemd\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.798840 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.798769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-run\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.798840 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.798789    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-tmp\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.798840 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.798831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-modprobe-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.798998 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.798919    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-registration-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.798998 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.798948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-lib-modules\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.798998 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.798973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-tuned\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.799093 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-socket-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.799093 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799038    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzqc\" (UniqueName: \"kubernetes.io/projected/5b07433f-bfec-4e99-97e4-1af1646c45ac-kube-api-access-wbzqc\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.799093 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysconfig\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.799183 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-host\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.799183 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799134    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvfj2\" (UniqueName: \"kubernetes.io/projected/8d5bf13e-c653-45d9-85f2-a878a8de5236-kube-api-access-bvfj2\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.799183 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f74d5816-8e0a-4f49-83bd-5cd907a57cab-host\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:41:29.799303 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799181    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxnl\" (UniqueName: \"kubernetes.io/projected/f74d5816-8e0a-4f49-83bd-5cd907a57cab-kube-api-access-xsxnl\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:41:29.799303 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.799303 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-device-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.799303 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-kubernetes\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.799419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799362    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.799419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-sys\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.799419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-var-lib-kubelet\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.799419 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f74d5816-8e0a-4f49-83bd-5cd907a57cab-serviceca\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:41:29.799600 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.799600 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.799481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-sys-fs\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.800762 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.800749    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:29.803247 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.803228    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:29.805594 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.805578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:41:29.807966 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.807949    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:41:29.810578 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.810560    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.842663 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.842637    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 03:41:29.842751 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.842679    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qv2jj\""
Nov 01 03:41:29.842751 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.842679    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 03:41:29.842751 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.842696    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 03:41:29.844649 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.844633    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 03:41:29.844649 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.844645    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 03:41:29.844769 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.844650    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 03:41:29.844874 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.844860    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h4w6m\""
Nov 01 03:41:29.846580 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.846565    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:41:29.847166 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.847151    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 03:41:29.847205 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.847183    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tmt47\""
Nov 01 03:41:29.854493 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.854476    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 03:41:29.854563 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.854490    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 03:41:29.854563 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.854547    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 03:41:29.854563 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.854554    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 03:41:29.854676 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.854576    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-prgzf\""
Nov 01 03:41:29.859052 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.859039    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 03:41:29.859291 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.859275    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-v2zmn\""
Nov 01 03:41:29.859329 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.859306    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 03:41:29.861763 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.861742    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 03:41:29.861868 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.861828    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jjmfj\""
Nov 01 03:41:29.861868 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.861861    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 03:41:29.863814 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.863794    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 03:41:29.864323 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.864306    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 03:41:29.864368 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.864310    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zn4h5\""
Nov 01 03:41:29.868666 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.868650    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7tm86\""
Nov 01 03:41:29.888917 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.888881    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 03:41:29.895010 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.894986    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 03:41:29.895010 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.895008    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 03:41:29.895132 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.895011    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:41:29.898798 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.898758    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 03:41:29.898962 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.898913    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 03:41:29.898962 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.898952    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 03:41:29.899118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.898986    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 03:41:29.899118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.899018    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dc6ls\""
Nov 01 03:41:29.899118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.899023    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 03:41:29.899212 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.899134    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 03:41:29.899212 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.899146    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 03:41:29.899212 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.899190    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b4qv6"
Nov 01 03:41:29.900346 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-multus\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.900391 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-system-cni-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:29.900391 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-lib-modules\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.900473 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c80204c-9fa6-4c25-a1e6-737c7f217976-tmp-dir\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:41:29.900533 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900495    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:29.900533 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-config\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.900620 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-script-lib\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.900620 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-socket-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.900752 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-dbus\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:29.900752 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900690    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-lib-modules\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.900752 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fllm\" (UniqueName: \"kubernetes.io/projected/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-kube-api-access-4fllm\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.900752 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.900752 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-var-lib-kubelet\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.900977 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900757    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-socket-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.900977 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900764    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flxck\" (UniqueName: \"kubernetes.io/projected/b496de82-bd77-42c5-8a29-e38514f99c5d-kube-api-access-flxck\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:29.900977 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900809    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.900977 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900812    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.900977 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900818    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-var-lib-kubelet\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.900977 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba9103b6-e703-48e0-9276-aec7726fcd36-host-slash\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:41:29.900977 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-systemd-units\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.900977 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-slash\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.900977 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.900977 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.900969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-sys-fs\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.901351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901015    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-sys-fs\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.901351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-systemd\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.901351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-kubelet\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.901351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-kubelet\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.901351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901139    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-netd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.901351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901167    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b92hs\" (UniqueName: \"kubernetes.io/projected/9c80204c-9fa6-4c25-a1e6-737c7f217976-kube-api-access-b92hs\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:41:29.901351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901182    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-systemd\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.901351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.901351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzqc\" (UniqueName: \"kubernetes.io/projected/5b07433f-bfec-4e99-97e4-1af1646c45ac-kube-api-access-wbzqc\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.901351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysconfig\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-host\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae127ade-3e32-4aa3-a528-324aa704bcdf-agent-certs\") pod \"konnectivity-agent-mm8h5\" (UID: \"ae127ade-3e32-4aa3-a528-324aa704bcdf\") " pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-cni-binary-copy\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901440    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysconfig\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cnibin\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901490    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-host\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-etc-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901544    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-node-log\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxnl\" (UniqueName: \"kubernetes.io/projected/f74d5816-8e0a-4f49-83bd-5cd907a57cab-kube-api-access-xsxnl\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-kubernetes\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-sys\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-tmp\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-system-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htgqq\" (UniqueName: \"kubernetes.io/projected/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-kube-api-access-htgqq\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901707    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-kubernetes\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.901813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-var-lib-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901752    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-sys\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-conf\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-run\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-netns\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-bin\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901881    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-conf\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901919    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901957    2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901970    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-run\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.901984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c80204c-9fa6-4c25-a1e6-737c7f217976-hosts-file\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvfj2\" (UniqueName: \"kubernetes.io/projected/8d5bf13e-c653-45d9-85f2-a878a8de5236-kube-api-access-bvfj2\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-kubelet-config\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902246    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-multus-certs\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-netns\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-ovn\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f74d5816-8e0a-4f49-83bd-5cd907a57cab-host\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:41:29.902597 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902373    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f74d5816-8e0a-4f49-83bd-5cd907a57cab-host\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-device-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902457    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-device-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902475    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-bin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-etc-kubernetes\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82ht\" (UniqueName: \"kubernetes.io/projected/20c62809-5ccf-419d-afae-8d140f67309d-kube-api-access-g82ht\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba9103b6-e703-48e0-9276-aec7726fcd36-iptables-alerter-script\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902645    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-cnibin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-systemd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-log-socket\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902757    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovn-node-metrics-cert\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902762    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-modprobe-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-registration-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.903416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902845    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-tuned\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902860    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-hostroot\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-conf-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902884    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-registration-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902898    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902920    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-modprobe-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpgbw\" (UniqueName: \"kubernetes.io/projected/ba9103b6-e703-48e0-9276-aec7726fcd36-kube-api-access-gpgbw\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.902960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.903040    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.903041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-socket-dir-parent\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.903082    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-k8s-cni-cncf-io\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.903103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-multus-daemon-config\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.903119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-env-overrides\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.903149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f74d5816-8e0a-4f49-83bd-5cd907a57cab-serviceca\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.903169    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcbx\" (UniqueName: \"kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx\") pod \"network-check-target-hbgqw\" (UID: \"42ba278c-e933-4575-872b-3420353c400a\") " pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.903200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae127ade-3e32-4aa3-a528-324aa704bcdf-konnectivity-ca\") pod \"konnectivity-agent-mm8h5\" (UID: \"ae127ade-3e32-4aa3-a528-324aa704bcdf\") " pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:29.903924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.903513    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f74d5816-8e0a-4f49-83bd-5cd907a57cab-serviceca\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:41:29.905409 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.905391    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-tmp\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.905452 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.905435    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-tuned\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.969515 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.969484    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvfj2\" (UniqueName: \"kubernetes.io/projected/8d5bf13e-c653-45d9-85f2-a878a8de5236-kube-api-access-bvfj2\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:29.969515 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.969501    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzqc\" (UniqueName: \"kubernetes.io/projected/5b07433f-bfec-4e99-97e4-1af1646c45ac-kube-api-access-wbzqc\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:29.969720 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:29.969597    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxnl\" (UniqueName: \"kubernetes.io/projected/f74d5816-8e0a-4f49-83bd-5cd907a57cab-kube-api-access-xsxnl\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:41:30.004071 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae127ade-3e32-4aa3-a528-324aa704bcdf-agent-certs\") pod \"konnectivity-agent-mm8h5\" (UID: \"ae127ade-3e32-4aa3-a528-324aa704bcdf\") " pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:30.004169 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004073    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-cni-binary-copy\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.004169 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cnibin\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.004169 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-etc-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004169 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-node-log\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004173    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cnibin\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.004350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004181    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-etc-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004190    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-node-log\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-system-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.004350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.004350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htgqq\" (UniqueName: \"kubernetes.io/projected/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-kube-api-access-htgqq\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.004350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-var-lib-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-netns\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.004350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004304    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-system-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.004350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-bin\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004351    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-bin\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004383    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-var-lib-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c80204c-9fa6-4c25-a1e6-737c7f217976-hosts-file\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004420    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004446    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-netns\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004496    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c80204c-9fa6-4c25-a1e6-737c7f217976-hosts-file\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.004519    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004566    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-kubelet-config\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.004585    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs podName:b496de82-bd77-42c5-8a29-e38514f99c5d nodeName:}" failed. No retries permitted until 2026-11-01 03:41:30.504564446 +0000 UTC m=+3.207796296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs") pod "network-metrics-daemon-vqrc7" (UID: "b496de82-bd77-42c5-8a29-e38514f99c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-kubelet-config\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-multus-certs\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-netns\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-ovn\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004679    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-cni-binary-copy\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:30.004904 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-bin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004734    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-netns\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-etc-kubernetes\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004785    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-ovn\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.004791    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82ht\" (UniqueName: \"kubernetes.io/projected/20c62809-5ccf-419d-afae-8d140f67309d-kube-api-access-g82ht\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004734    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-multus-certs\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004835    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004848    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-etc-kubernetes\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004761    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-bin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.004872    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret podName:b3f80953-074f-455a-9c88-6b3154926050 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:30.504855577 +0000 UTC m=+3.208087421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret") pod "global-pull-secret-syncer-zdw9t" (UID: "b3f80953-074f-455a-9c88-6b3154926050") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba9103b6-e703-48e0-9276-aec7726fcd36-iptables-alerter-script\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-cnibin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004952    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.004972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-systemd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005001    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-cnibin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.005661 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005014    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-log-socket\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovn-node-metrics-cert\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-hostroot\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005096    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-log-socket\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005054    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-systemd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005116    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-conf-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005133    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-hostroot\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005158    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-conf-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpgbw\" (UniqueName: \"kubernetes.io/projected/ba9103b6-e703-48e0-9276-aec7726fcd36-kube-api-access-gpgbw\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-socket-dir-parent\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005243    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-k8s-cni-cncf-io\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-multus-daemon-config\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-env-overrides\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcbx\" (UniqueName: \"kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx\") pod \"network-check-target-hbgqw\" (UID: \"42ba278c-e933-4575-872b-3420353c400a\") " pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005347    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-socket-dir-parent\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae127ade-3e32-4aa3-a528-324aa704bcdf-konnectivity-ca\") pod \"konnectivity-agent-mm8h5\" (UID: \"ae127ade-3e32-4aa3-a528-324aa704bcdf\") " pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:30.006358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-multus\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-system-cni-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005444    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-k8s-cni-cncf-io\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005452    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c80204c-9fa6-4c25-a1e6-737c7f217976-tmp-dir\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-config\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-script-lib\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-dbus\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fllm\" (UniqueName: \"kubernetes.io/projected/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-kube-api-access-4fllm\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxck\" (UniqueName: \"kubernetes.io/projected/b496de82-bd77-42c5-8a29-e38514f99c5d-kube-api-access-flxck\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba9103b6-e703-48e0-9276-aec7726fcd36-host-slash\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-systemd-units\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-slash\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-kubelet\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-kubelet\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.006933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005874    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005885    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-netd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b92hs\" (UniqueName: \"kubernetes.io/projected/9c80204c-9fa6-4c25-a1e6-737c7f217976-kube-api-access-b92hs\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005948    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.005991    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-dbus\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.006041    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba9103b6-e703-48e0-9276-aec7726fcd36-host-slash\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.006042    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.006071    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-multus\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.006125    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-slash\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.006125    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-systemd-units\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.006190    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-netd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.006341    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba9103b6-e703-48e0-9276-aec7726fcd36-iptables-alerter-script\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.006568    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.006620    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.006649    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-env-overrides\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.006745    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-multus-daemon-config\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.007031    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae127ade-3e32-4aa3-a528-324aa704bcdf-konnectivity-ca\") pod \"konnectivity-agent-mm8h5\" (UID: \"ae127ade-3e32-4aa3-a528-324aa704bcdf\") " pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:30.007549 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.007047    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae127ade-3e32-4aa3-a528-324aa704bcdf-agent-certs\") pod \"konnectivity-agent-mm8h5\" (UID: \"ae127ade-3e32-4aa3-a528-324aa704bcdf\") " pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:30.008024 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.007074    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.008024 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.007084    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-config\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.008024 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.007127    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-kubelet\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.008024 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.007130    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-script-lib\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.008024 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.007153    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-system-cni-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.008024 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.007204    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-kubelet\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.008024 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.007384    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c80204c-9fa6-4c25-a1e6-737c7f217976-tmp-dir\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:41:30.008024 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.007428    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovn-node-metrics-cert\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.072448 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.072411    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82ht\" (UniqueName: \"kubernetes.io/projected/20c62809-5ccf-419d-afae-8d140f67309d-kube-api-access-g82ht\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.072642 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.072616    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgqq\" (UniqueName: \"kubernetes.io/projected/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-kube-api-access-htgqq\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.072731 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.072717    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpgbw\" (UniqueName: \"kubernetes.io/projected/ba9103b6-e703-48e0-9276-aec7726fcd36-kube-api-access-gpgbw\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:41:30.073868 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.073852    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:41:30.073919 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.073871    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:41:30.073919 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.073882    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4gcbx for pod openshift-network-diagnostics/network-check-target-hbgqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:30.073975 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.073929    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx podName:42ba278c-e933-4575-872b-3420353c400a nodeName:}" failed. No retries permitted until 2026-11-01 03:41:30.573916406 +0000 UTC m=+3.277148260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4gcbx" (UniqueName: "kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx") pod "network-check-target-hbgqw" (UID: "42ba278c-e933-4575-872b-3420353c400a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:30.078548 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.078527    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b92hs\" (UniqueName: \"kubernetes.io/projected/9c80204c-9fa6-4c25-a1e6-737c7f217976-kube-api-access-b92hs\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:41:30.078642 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.078620    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxck\" (UniqueName: \"kubernetes.io/projected/b496de82-bd77-42c5-8a29-e38514f99c5d-kube-api-access-flxck\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:30.078946 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.078930    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fllm\" (UniqueName: \"kubernetes.io/projected/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-kube-api-access-4fllm\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.095067 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.095047    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:41:30.105469 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:30.105422    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74d5816_8e0a_4f49_83bd_5cd907a57cab.slice/crio-57ff89a064810529b94be1b6ba9e0dd51706a0c0d1f572b52d1b3988334a5df4 WatchSource:0}: Error finding container 57ff89a064810529b94be1b6ba9e0dd51706a0c0d1f572b52d1b3988334a5df4: Status 404 returned error can't find the container with id 57ff89a064810529b94be1b6ba9e0dd51706a0c0d1f572b52d1b3988334a5df4
Nov 01 03:41:30.106576 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.106552    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:41:30.111628 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.111612    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:41:30.114404 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.114340    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m7dq2"
Nov 01 03:41:30.114768 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:30.114747    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b07433f_bfec_4e99_97e4_1af1646c45ac.slice/crio-2508a5e206e9538f252ec6fe28ebb3026b5f5208ff47d6befd52c57ac618fe80 WatchSource:0}: Error finding container 2508a5e206e9538f252ec6fe28ebb3026b5f5208ff47d6befd52c57ac618fe80: Status 404 returned error can't find the container with id 2508a5e206e9538f252ec6fe28ebb3026b5f5208ff47d6befd52c57ac618fe80
Nov 01 03:41:30.120594 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.120575    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:30.121051 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:30.121032    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5bf13e_c653_45d9_85f2_a878a8de5236.slice/crio-d1964055aaa699035af17544824a086210790bf6e1ffab4889809a8f4b431753 WatchSource:0}: Error finding container d1964055aaa699035af17544824a086210790bf6e1ffab4889809a8f4b431753: Status 404 returned error can't find the container with id d1964055aaa699035af17544824a086210790bf6e1ffab4889809a8f4b431753
Nov 01 03:41:30.125009 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:30.124974    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c62809_5ccf_419d_afae_8d140f67309d.slice/crio-5333dc8a413658f507ade37494d3b35e2d7b13b003e407dcceb2d358e579cbcb WatchSource:0}: Error finding container 5333dc8a413658f507ade37494d3b35e2d7b13b003e407dcceb2d358e579cbcb: Status 404 returned error can't find the container with id 5333dc8a413658f507ade37494d3b35e2d7b13b003e407dcceb2d358e579cbcb
Nov 01 03:41:30.130903 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:30.130883    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae127ade_3e32_4aa3_a528_324aa704bcdf.slice/crio-b6f1b2cd380df1338c03abfd519930a57f929e67f5dc36b6ee3a5fdaa693207f WatchSource:0}: Error finding container b6f1b2cd380df1338c03abfd519930a57f929e67f5dc36b6ee3a5fdaa693207f: Status 404 returned error can't find the container with id b6f1b2cd380df1338c03abfd519930a57f929e67f5dc36b6ee3a5fdaa693207f
Nov 01 03:41:30.137773 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.137756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:41:30.139978 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.139941    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:41:30.143764 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.143750    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:41:30.148076 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:30.148059    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c80204c_9fa6_4c25_a1e6_737c7f217976.slice/crio-d2739256e3f744805bb0bdff60b5cd89a58785399ab8cf2bc66bdedd3cf016ae WatchSource:0}: Error finding container d2739256e3f744805bb0bdff60b5cd89a58785399ab8cf2bc66bdedd3cf016ae: Status 404 returned error can't find the container with id d2739256e3f744805bb0bdff60b5cd89a58785399ab8cf2bc66bdedd3cf016ae
Nov 01 03:41:30.151397 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:30.151378    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9103b6_e703_48e0_9276_aec7726fcd36.slice/crio-acdfc83ea304eb9b14652fcaedebc62bcc8c79257d72795c3dccd900e684c119 WatchSource:0}: Error finding container acdfc83ea304eb9b14652fcaedebc62bcc8c79257d72795c3dccd900e684c119: Status 404 returned error can't find the container with id acdfc83ea304eb9b14652fcaedebc62bcc8c79257d72795c3dccd900e684c119
Nov 01 03:41:30.153657 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:30.153639    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600cdad1_59e2_4bdd_b73e_cb475f7c9cc5.slice/crio-e6a1497a78002a8af458579b2c7a0bfba8207fe50ab4a01890986e51cbee0832 WatchSource:0}: Error finding container e6a1497a78002a8af458579b2c7a0bfba8207fe50ab4a01890986e51cbee0832: Status 404 returned error can't find the container with id e6a1497a78002a8af458579b2c7a0bfba8207fe50ab4a01890986e51cbee0832
Nov 01 03:41:30.161963 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.161948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:30.170696 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:30.170675    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf065d74a_fa1d_41f2_b4a9_6db03bd53a71.slice/crio-f249a50d3760b1f2cd241f459207c71c50bf8e97cecc190068a2042afaf009a2 WatchSource:0}: Error finding container f249a50d3760b1f2cd241f459207c71c50bf8e97cecc190068a2042afaf009a2: Status 404 returned error can't find the container with id f249a50d3760b1f2cd241f459207c71c50bf8e97cecc190068a2042afaf009a2
Nov 01 03:41:30.508715 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.508638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:30.508715 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.508679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:30.508879 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.508793    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:30.508879 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.508795    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:30.508879 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.508845    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret podName:b3f80953-074f-455a-9c88-6b3154926050 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:31.508829503 +0000 UTC m=+4.212061345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret") pod "global-pull-secret-syncer-zdw9t" (UID: "b3f80953-074f-455a-9c88-6b3154926050") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:30.508879 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.508860    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs podName:b496de82-bd77-42c5-8a29-e38514f99c5d nodeName:}" failed. No retries permitted until 2026-11-01 03:41:31.508853607 +0000 UTC m=+4.212085439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs") pod "network-metrics-daemon-vqrc7" (UID: "b496de82-bd77-42c5-8a29-e38514f99c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:30.609908 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.609874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcbx\" (UniqueName: \"kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx\") pod \"network-check-target-hbgqw\" (UID: \"42ba278c-e933-4575-872b-3420353c400a\") " pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:30.610057 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.610037    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:41:30.610116 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.610061    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:41:30.610116 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.610073    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4gcbx for pod openshift-network-diagnostics/network-check-target-hbgqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:30.610188 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:30.610124    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx podName:42ba278c-e933-4575-872b-3420353c400a nodeName:}" failed. No retries permitted until 2026-11-01 03:41:31.610108016 +0000 UTC m=+4.313339861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4gcbx" (UniqueName: "kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx") pod "network-check-target-hbgqw" (UID: "42ba278c-e933-4575-872b-3420353c400a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:30.758390 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.758354    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g9vql" event={"ID":"ba9103b6-e703-48e0-9276-aec7726fcd36","Type":"ContainerStarted","Data":"acdfc83ea304eb9b14652fcaedebc62bcc8c79257d72795c3dccd900e684c119"}
Nov 01 03:41:30.759529 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.759430    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mm8h5" event={"ID":"ae127ade-3e32-4aa3-a528-324aa704bcdf","Type":"ContainerStarted","Data":"b6f1b2cd380df1338c03abfd519930a57f929e67f5dc36b6ee3a5fdaa693207f"}
Nov 01 03:41:30.760538 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.760498    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m7dq2" event={"ID":"20c62809-5ccf-419d-afae-8d140f67309d","Type":"ContainerStarted","Data":"5333dc8a413658f507ade37494d3b35e2d7b13b003e407dcceb2d358e579cbcb"}
Nov 01 03:41:30.761605 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.761576    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv" event={"ID":"5b07433f-bfec-4e99-97e4-1af1646c45ac","Type":"ContainerStarted","Data":"2508a5e206e9538f252ec6fe28ebb3026b5f5208ff47d6befd52c57ac618fe80"}
Nov 01 03:41:30.762676 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.762655    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8tdn" event={"ID":"9c80204c-9fa6-4c25-a1e6-737c7f217976","Type":"ContainerStarted","Data":"d2739256e3f744805bb0bdff60b5cd89a58785399ab8cf2bc66bdedd3cf016ae"}
Nov 01 03:41:30.763824 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.763804    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kwns5" event={"ID":"8d5bf13e-c653-45d9-85f2-a878a8de5236","Type":"ContainerStarted","Data":"d1964055aaa699035af17544824a086210790bf6e1ffab4889809a8f4b431753"}
Nov 01 03:41:30.764836 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.764816    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fwzpl" event={"ID":"f74d5816-8e0a-4f49-83bd-5cd907a57cab","Type":"ContainerStarted","Data":"57ff89a064810529b94be1b6ba9e0dd51706a0c0d1f572b52d1b3988334a5df4"}
Nov 01 03:41:30.765860 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.765841    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4" event={"ID":"f065d74a-fa1d-41f2-b4a9-6db03bd53a71","Type":"ContainerStarted","Data":"f249a50d3760b1f2cd241f459207c71c50bf8e97cecc190068a2042afaf009a2"}
Nov 01 03:41:30.766979 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.766961    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jz78s" event={"ID":"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5","Type":"ContainerStarted","Data":"e6a1497a78002a8af458579b2c7a0bfba8207fe50ab4a01890986e51cbee0832"}
Nov 01 03:41:30.902150 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.902096    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:36:30 +0000 UTC" deadline="2028-05-19 14:41:13.021754913 +0000 UTC"
Nov 01 03:41:30.902150 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:30.902141    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13570h59m42.119617987s"
Nov 01 03:41:31.517704 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:31.517128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:31.517704 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:31.517261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:31.517704 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:31.517437    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:31.517704 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:31.517524    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs podName:b496de82-bd77-42c5-8a29-e38514f99c5d nodeName:}" failed. No retries permitted until 2026-11-01 03:41:33.51750318 +0000 UTC m=+6.220735030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs") pod "network-metrics-daemon-vqrc7" (UID: "b496de82-bd77-42c5-8a29-e38514f99c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:31.517704 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:31.517625    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:31.517704 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:31.517667    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret podName:b3f80953-074f-455a-9c88-6b3154926050 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:33.517654227 +0000 UTC m=+6.220886073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret") pod "global-pull-secret-syncer-zdw9t" (UID: "b3f80953-074f-455a-9c88-6b3154926050") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:31.618276 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:31.618226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcbx\" (UniqueName: \"kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx\") pod \"network-check-target-hbgqw\" (UID: \"42ba278c-e933-4575-872b-3420353c400a\") " pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:31.618461 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:31.618390    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:41:31.618461 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:31.618413    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:41:31.618461 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:31.618426    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4gcbx for pod openshift-network-diagnostics/network-check-target-hbgqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:31.618659 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:31.618577    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx podName:42ba278c-e933-4575-872b-3420353c400a nodeName:}" failed. No retries permitted until 2026-11-01 03:41:33.618556191 +0000 UTC m=+6.321788021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4gcbx" (UniqueName: "kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx") pod "network-check-target-hbgqw" (UID: "42ba278c-e933-4575-872b-3420353c400a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:31.749540 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:31.748447    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:31.749540 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:31.748604    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:31.749540 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:31.748708    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:31.749540 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:31.748810    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:31.749540 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:31.748870    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:31.749540 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:31.748940    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:31.903067 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:31.902963    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:36:30 +0000 UTC" deadline="2028-05-19 07:32:34.909225366 +0000 UTC"
Nov 01 03:41:31.903067 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:31.903020    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13563h51m3.006211321s"
Nov 01 03:41:33.535329 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:33.534880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:33.535329 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:33.534932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:33.535329 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:33.535100    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:33.535329 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:33.535166    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret podName:b3f80953-074f-455a-9c88-6b3154926050 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:37.535145379 +0000 UTC m=+10.238377229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret") pod "global-pull-secret-syncer-zdw9t" (UID: "b3f80953-074f-455a-9c88-6b3154926050") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:33.535329 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:33.535241    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:33.535329 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:33.535284    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs podName:b496de82-bd77-42c5-8a29-e38514f99c5d nodeName:}" failed. No retries permitted until 2026-11-01 03:41:37.535270489 +0000 UTC m=+10.238502336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs") pod "network-metrics-daemon-vqrc7" (UID: "b496de82-bd77-42c5-8a29-e38514f99c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:33.636195 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:33.635906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcbx\" (UniqueName: \"kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx\") pod \"network-check-target-hbgqw\" (UID: \"42ba278c-e933-4575-872b-3420353c400a\") " pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:33.636195 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:33.636062    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:41:33.636195 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:33.636082    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:41:33.636195 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:33.636094    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4gcbx for pod openshift-network-diagnostics/network-check-target-hbgqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:33.636195 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:33.636151    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx podName:42ba278c-e933-4575-872b-3420353c400a nodeName:}" failed. No retries permitted until 2026-11-01 03:41:37.636131417 +0000 UTC m=+10.339363265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4gcbx" (UniqueName: "kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx") pod "network-check-target-hbgqw" (UID: "42ba278c-e933-4575-872b-3420353c400a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:33.755521 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:33.755486    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:33.755690 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:33.755630    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:33.755766 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:33.755740    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:33.756097 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:33.755828    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:33.756097 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:33.755884    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:33.756097 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:33.755957    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:34.785473 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:34.785360    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5d0b123ba86397e4d3cfebb9bec2811" containerID="b14cff8f77ef0faea1540e809bb47a321741da0e56303fdfd76e620ee64d14fd" exitCode=0
Nov 01 03:41:34.785473 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:34.785451    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal" event={"ID":"a5d0b123ba86397e4d3cfebb9bec2811","Type":"ContainerDied","Data":"b14cff8f77ef0faea1540e809bb47a321741da0e56303fdfd76e620ee64d14fd"}
Nov 01 03:41:34.796468 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:34.796377    2583 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="a13c915b9c1bdc801cade534b923931f0f786769bf29b5917abb6181daff0217" exitCode=0
Nov 01 03:41:34.796468 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:34.796427    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jz78s" event={"ID":"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5","Type":"ContainerDied","Data":"a13c915b9c1bdc801cade534b923931f0f786769bf29b5917abb6181daff0217"}
Nov 01 03:41:35.753838 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:35.753008    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:35.753838 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:35.753139    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:35.753838 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:35.753441    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:35.753838 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:35.753562    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:35.753838 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:35.753618    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:35.753838 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:35.753687    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:35.802690 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:35.801852    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal" event={"ID":"349f3d418ab214cf804b3c489f47ad32","Type":"ContainerStarted","Data":"69308688dc7db7d64687dba04cbe3734d217211b1f39d4c4bb33c4d6afc617b7"}
Nov 01 03:41:35.811856 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:35.811822    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal" event={"ID":"a5d0b123ba86397e4d3cfebb9bec2811","Type":"ContainerStarted","Data":"0587ceef499b0376606eb26e6b4bd8f3d5298295978e30cb7a7d88e22ba6652e"}
Nov 01 03:41:35.818033 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:35.817965    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mm8h5" event={"ID":"ae127ade-3e32-4aa3-a528-324aa704bcdf","Type":"ContainerStarted","Data":"f9c64d774d7ec938dae46d979827ec15691d89042840a083e31cfa66284b6883"}
Nov 01 03:41:36.061704 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:36.061639    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal" podStartSLOduration=6.061616492 podStartE2EDuration="6.061616492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:35.931675601 +0000 UTC m=+8.634907455" watchObservedRunningTime="2026-11-01 03:41:36.061616492 +0000 UTC m=+8.764848344"
Nov 01 03:41:36.194053 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:36.193986    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mm8h5" podStartSLOduration=1.753405318 podStartE2EDuration="7.193965663s" totalImagesPullingTime="5.440560345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:36.062231969 +0000 UTC m=+8.765463822" watchObservedRunningTime="2026-11-01 03:41:36.193965663 +0000 UTC m=+8.897197520"
Nov 01 03:41:37.575806 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:37.575759    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:37.575806 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:37.575811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:37.576457 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:37.576011    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:37.576457 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:37.576087    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret podName:b3f80953-074f-455a-9c88-6b3154926050 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:45.576066773 +0000 UTC m=+18.279298620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret") pod "global-pull-secret-syncer-zdw9t" (UID: "b3f80953-074f-455a-9c88-6b3154926050") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:37.576457 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:37.576161    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:37.576457 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:37.576198    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs podName:b496de82-bd77-42c5-8a29-e38514f99c5d nodeName:}" failed. No retries permitted until 2026-11-01 03:41:45.576186831 +0000 UTC m=+18.279418663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs") pod "network-metrics-daemon-vqrc7" (UID: "b496de82-bd77-42c5-8a29-e38514f99c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:37.676848 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:37.676777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcbx\" (UniqueName: \"kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx\") pod \"network-check-target-hbgqw\" (UID: \"42ba278c-e933-4575-872b-3420353c400a\") " pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:37.677046 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:37.676942    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:41:37.677046 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:37.676962    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:41:37.677046 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:37.676975    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4gcbx for pod openshift-network-diagnostics/network-check-target-hbgqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:37.677046 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:37.677031    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx podName:42ba278c-e933-4575-872b-3420353c400a nodeName:}" failed. No retries permitted until 2026-11-01 03:41:45.677012847 +0000 UTC m=+18.380244690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4gcbx" (UniqueName: "kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx") pod "network-check-target-hbgqw" (UID: "42ba278c-e933-4575-872b-3420353c400a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:37.759102 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:37.759066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:37.759296 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:37.759267    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:37.763599 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:37.759363    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:37.763599 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:37.759516    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:37.763599 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:37.762848    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:37.763599 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:37.763061    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:37.830660 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:37.830468    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv" event={"ID":"5b07433f-bfec-4e99-97e4-1af1646c45ac","Type":"ContainerStarted","Data":"453dc468353d9b33498f1ead10c350ba0c941f150da81cb7707c18ac3dacd905"}
Nov 01 03:41:37.832903 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:37.832389    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fwzpl" event={"ID":"f74d5816-8e0a-4f49-83bd-5cd907a57cab","Type":"ContainerStarted","Data":"9eab43a12236bd3d1fa31c273a78e94b70e89a211d77652361020899a9f6bf2d"}
Nov 01 03:41:37.960195 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:37.960133    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal" podStartSLOduration=7.960113927 podStartE2EDuration="7.960113927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:36.194573281 +0000 UTC m=+8.897805134" watchObservedRunningTime="2026-11-01 03:41:37.960113927 +0000 UTC m=+10.663345782"
Nov 01 03:41:38.021000 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:38.020879    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:38.022259 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:38.022057    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:38.152559 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:38.152116    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fwzpl" podStartSLOduration=3.640267987 podStartE2EDuration="10.152094593s" totalImagesPullingTime="6.511826606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:37.960951278 +0000 UTC m=+10.664183131" watchObservedRunningTime="2026-11-01 03:41:38.152094593 +0000 UTC m=+10.855326449"
Nov 01 03:41:38.835810 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:38.835730    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:38.837469 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:38.837255    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:41:39.636049 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:39.636023    2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 03:41:39.755736 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:39.755432    2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T03:41:39.636045626Z","UUID":"4a13de08-b5ab-4b72-b780-5641d95d8add","Handler":null,"Name":"","Endpoint":""}
Nov 01 03:41:39.757230 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:39.756826    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:39.757230 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:39.756956    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:39.757230 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:39.757034    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:39.757230 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:39.757132    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:39.757230 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:39.757178    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:39.757667 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:39.757243    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:39.760699 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:39.760676    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 03:41:39.760807 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:39.760712    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 03:41:39.842106 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:39.841644    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m7dq2" event={"ID":"20c62809-5ccf-419d-afae-8d140f67309d","Type":"ContainerStarted","Data":"1ad58838deb251cdcaf2e52eaaaea2e433e3cbe9cceeaf4633dff0b32c428055"}
Nov 01 03:41:39.845036 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:39.844972    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv" event={"ID":"5b07433f-bfec-4e99-97e4-1af1646c45ac","Type":"ContainerStarted","Data":"5d028734a01535e3f653134ac62879ddda354ec714600c3edf65fc3fc3735e14"}
Nov 01 03:41:39.978316 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:39.978266    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-m7dq2" podStartSLOduration=1.7493204549999999 podStartE2EDuration="10.978250833s" totalImagesPullingTime="9.228930378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:39.977977443 +0000 UTC m=+12.681209294" watchObservedRunningTime="2026-11-01 03:41:39.978250833 +0000 UTC m=+12.681482681"
Nov 01 03:41:40.849602 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:40.849407    2583 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="8d25a9fe6dd518faa4dc04518ca942535d39e86c64720d80eea66e636cc2d9a2" exitCode=0
Nov 01 03:41:40.850235 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:40.849498    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jz78s" event={"ID":"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5","Type":"ContainerDied","Data":"8d25a9fe6dd518faa4dc04518ca942535d39e86c64720d80eea66e636cc2d9a2"}
Nov 01 03:41:40.852451 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:40.852404    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8tdn" event={"ID":"9c80204c-9fa6-4c25-a1e6-737c7f217976","Type":"ContainerStarted","Data":"b01146202d7ec2f7295d04e2d81fa2b2cca1784ae61d952854fd59246f9c2f57"}
Nov 01 03:41:41.097422 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:41.097350    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-t8tdn" podStartSLOduration=2.5800933219999997 podStartE2EDuration="12.097332972s" totalImagesPullingTime="9.51723965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:41.097031983 +0000 UTC m=+13.800263834" watchObservedRunningTime="2026-11-01 03:41:41.097332972 +0000 UTC m=+13.800564822"
Nov 01 03:41:41.747658 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:41.747626    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:41.747834 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:41.747665    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:41.747834 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:41.747749    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:41.747834 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:41.747797    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:41.747834 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:41.747812    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:41.748029 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:41.747922    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:41.855685 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:41.855646    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g9vql" event={"ID":"ba9103b6-e703-48e0-9276-aec7726fcd36","Type":"ContainerStarted","Data":"d1db2bab159d32b65cb5251b76c3088c3392707eb5aba789b5a4cb1a350c59d0"}
Nov 01 03:41:41.974987 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:41.974927    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g9vql" podStartSLOduration=3.248712263 podStartE2EDuration="12.974908647s" totalImagesPullingTime="9.726196384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:41.974155416 +0000 UTC m=+14.677387265" watchObservedRunningTime="2026-11-01 03:41:41.974908647 +0000 UTC m=+14.678140497"
Nov 01 03:41:42.860647 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:42.860572    2583 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="7720e617a62305528e11d2cc2bd713b4d055a5c8ab954c0716e4e45b79a092cd" exitCode=0
Nov 01 03:41:42.861246 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:42.860657    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jz78s" event={"ID":"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5","Type":"ContainerDied","Data":"7720e617a62305528e11d2cc2bd713b4d055a5c8ab954c0716e4e45b79a092cd"}
Nov 01 03:41:42.863720 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:42.863694    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv" event={"ID":"5b07433f-bfec-4e99-97e4-1af1646c45ac","Type":"ContainerStarted","Data":"aada0da99ef03e78cc51a87b5973f84d8468c72cd182c1f85060a47dfd66e7b6"}
Nov 01 03:41:42.865200 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:42.865174    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kwns5" event={"ID":"8d5bf13e-c653-45d9-85f2-a878a8de5236","Type":"ContainerStarted","Data":"b659b46d1a62ad9dd83218075072a60c609dd16967542c433cb47d7b0efa74bd"}
Nov 01 03:41:43.109791 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:43.109738    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv" podStartSLOduration=3.595979736 podStartE2EDuration="15.109723467s" totalImagesPullingTime="11.513743731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:43.10944508 +0000 UTC m=+15.812676932" watchObservedRunningTime="2026-11-01 03:41:43.109723467 +0000 UTC m=+15.812955315"
Nov 01 03:41:43.230937 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:43.230886    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kwns5" podStartSLOduration=3.335660201 podStartE2EDuration="15.230867965s" totalImagesPullingTime="11.895207764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:43.230753582 +0000 UTC m=+15.933985432" watchObservedRunningTime="2026-11-01 03:41:43.230867965 +0000 UTC m=+15.934099814"
Nov 01 03:41:43.750974 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:43.750943    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:43.751144 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:43.751008    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:43.751144 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:43.751109    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:43.751144 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:43.751121    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:43.751267 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:43.751197    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:43.751301 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:43.751271    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:44.871612 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:44.871331    2583 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="c74679b656c9f86e19a1f18d29b0a85b3febe357962efcaa39c6fe310357709e" exitCode=0
Nov 01 03:41:44.871612 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:44.871485    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jz78s" event={"ID":"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5","Type":"ContainerDied","Data":"c74679b656c9f86e19a1f18d29b0a85b3febe357962efcaa39c6fe310357709e"}
Nov 01 03:41:45.632755 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:45.632716    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:45.632942 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:45.632769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:45.632942 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:45.632832    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:45.632942 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:45.632885    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:45.632942 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:45.632897    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs podName:b496de82-bd77-42c5-8a29-e38514f99c5d nodeName:}" failed. No retries permitted until 2026-11-01 03:42:01.632877908 +0000 UTC m=+34.336109751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs") pod "network-metrics-daemon-vqrc7" (UID: "b496de82-bd77-42c5-8a29-e38514f99c5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:41:45.632942 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:45.632927    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret podName:b3f80953-074f-455a-9c88-6b3154926050 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:01.632912621 +0000 UTC m=+34.336144458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret") pod "global-pull-secret-syncer-zdw9t" (UID: "b3f80953-074f-455a-9c88-6b3154926050") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:41:45.733596 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:45.733559    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcbx\" (UniqueName: \"kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx\") pod \"network-check-target-hbgqw\" (UID: \"42ba278c-e933-4575-872b-3420353c400a\") " pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:45.733753 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:45.733736    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:41:45.733838 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:45.733763    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:41:45.733838 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:45.733783    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4gcbx for pod openshift-network-diagnostics/network-check-target-hbgqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:45.733910 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:45.733845    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx podName:42ba278c-e933-4575-872b-3420353c400a nodeName:}" failed. No retries permitted until 2026-11-01 03:42:01.733826852 +0000 UTC m=+34.437058684 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4gcbx" (UniqueName: "kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx") pod "network-check-target-hbgqw" (UID: "42ba278c-e933-4575-872b-3420353c400a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:41:45.747899 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:45.747725    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:45.747899 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:45.747746    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:45.747899 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:45.747838    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:45.748099 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:45.747965    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:45.748099 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:45.747993    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:45.748185 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:45.748103    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:46.880022 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:46.879825    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4" event={"ID":"f065d74a-fa1d-41f2-b4a9-6db03bd53a71","Type":"ContainerStarted","Data":"cc0ccb8f85717810952a5e2323de28ba32ea155d0294edfb8f5c1ed8a5aadd16"}
Nov 01 03:41:46.880403 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:46.880067    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4" event={"ID":"f065d74a-fa1d-41f2-b4a9-6db03bd53a71","Type":"ContainerStarted","Data":"8958cacbd71e70b7edf244323cedb53995584b65f5b3d8ad603fe49fd6052eae"}
Nov 01 03:41:46.880403 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:46.880101    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4" event={"ID":"f065d74a-fa1d-41f2-b4a9-6db03bd53a71","Type":"ContainerStarted","Data":"50a08f6951396f2ae15ad43d44288a096272aacf0c49c52677a2bc82c2506f18"}
Nov 01 03:41:46.880403 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:46.880129    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4" event={"ID":"f065d74a-fa1d-41f2-b4a9-6db03bd53a71","Type":"ContainerStarted","Data":"318c527bd7913e8449618fb9c782e6f2c32d319eef760ebffe4b8994305001e0"}
Nov 01 03:41:46.880403 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:46.880166    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4" event={"ID":"f065d74a-fa1d-41f2-b4a9-6db03bd53a71","Type":"ContainerStarted","Data":"c59a65436ba1aedbaf9e3ca03ab53fd50e7dd30d8f96c601cd33bc94f88c43f1"}
Nov 01 03:41:46.880403 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:46.880197    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4" event={"ID":"f065d74a-fa1d-41f2-b4a9-6db03bd53a71","Type":"ContainerStarted","Data":"883df17a3714bffa273d675f773bd218e83d66551a15e7427ac97a2104567a4e"}
Nov 01 03:41:47.748988 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:47.748953    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:47.749142 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:47.749037    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:47.749142 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:47.749120    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:47.749293 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:47.749270    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:47.749327 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:47.749279    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:47.749402 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:47.749385    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:47.884746 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:47.884716    2583 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="41a1ee6742f5cecac1bacbdcad96a512e0ca4c3b011189900e44c7020f4caf27" exitCode=0
Nov 01 03:41:47.884746 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:47.884750    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jz78s" event={"ID":"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5","Type":"ContainerDied","Data":"41a1ee6742f5cecac1bacbdcad96a512e0ca4c3b011189900e44c7020f4caf27"}
Nov 01 03:41:48.890047 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:48.890021    2583 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="aeb9a12b8b23e8f2fba77b1e99aa80c221125eb36d21a62e3131e6c19d6d1976" exitCode=0
Nov 01 03:41:48.890519 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:48.890068    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jz78s" event={"ID":"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5","Type":"ContainerDied","Data":"aeb9a12b8b23e8f2fba77b1e99aa80c221125eb36d21a62e3131e6c19d6d1976"}
Nov 01 03:41:49.748256 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:49.748227    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:49.748421 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:49.748278    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:49.748421 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:49.748299    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:49.748421 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:49.748347    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:49.748421 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:49.748391    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:49.748611 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:49.748459    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:49.895638 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:49.895605    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4" event={"ID":"f065d74a-fa1d-41f2-b4a9-6db03bd53a71","Type":"ContainerStarted","Data":"b98d1bcbd2cce8b3b0eb661c0608f797e68b944c1454e9c7989fbbabc7459d9a"}
Nov 01 03:41:49.898630 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:49.898608    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jz78s" event={"ID":"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5","Type":"ContainerStarted","Data":"4ca91b60a956e992cd250ac1ef49d6bb0f494a680465bd6b5c75cbb5038ee843"}
Nov 01 03:41:50.042390 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:50.042286    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jz78s" podStartSLOduration=6.091018219 podStartE2EDuration="21.042272573s" totalImagesPullingTime="14.951254354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:50.042000278 +0000 UTC m=+22.745232127" watchObservedRunningTime="2026-11-01 03:41:50.042272573 +0000 UTC m=+22.745504421"
Nov 01 03:41:51.747947 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:51.747778    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:51.748490 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:51.747786    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:51.748490 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:51.748028    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:51.748490 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:51.748085    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:51.748490 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:51.747848    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:51.748490 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:51.748150    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:51.906029 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:51.905992    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4" event={"ID":"f065d74a-fa1d-41f2-b4a9-6db03bd53a71","Type":"ContainerStarted","Data":"72ab4239a24d805115ddf645cb31c6e006bfc1b161249e180157c1bd0f220fa4"}
Nov 01 03:41:51.906310 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:51.906294    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:51.906380 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:51.906367    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:51.921729 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:51.921707    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:52.333992 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:52.333940    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4" podStartSLOduration=7.051843976 podStartE2EDuration="23.333927321s" totalImagesPullingTime="16.282083345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:52.199315386 +0000 UTC m=+24.902547236" watchObservedRunningTime="2026-11-01 03:41:52.333927321 +0000 UTC m=+25.037159170"
Nov 01 03:41:52.909003 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:52.908975    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:52.923194 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:52.923173    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:41:53.550527 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:53.550488    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zdw9t"]
Nov 01 03:41:53.550684 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:53.550598    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:53.550684 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:53.550674    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:53.553687 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:53.553665    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hbgqw"]
Nov 01 03:41:53.553817 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:53.553767    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:53.553855 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:53.553832    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:53.557680 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:53.557665    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vqrc7"]
Nov 01 03:41:53.557752 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:53.557741    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:53.557838 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:53.557823    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:55.747811 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:55.747771    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:55.747811 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:55.747787    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:55.748397 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:55.747887    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:55.748397 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:55.747935    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:55.748397 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:55.748032    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:55.748397 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:55.748112    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:57.748832 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:57.748809    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:57.749187 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:57.748900    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqrc7" podUID="b496de82-bd77-42c5-8a29-e38514f99c5d"
Nov 01 03:41:57.749187 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:57.748937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:57.749187 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:57.748998    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdw9t" podUID="b3f80953-074f-455a-9c88-6b3154926050"
Nov 01 03:41:57.749187 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:57.749033    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:57.749187 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:57.749099    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hbgqw" podUID="42ba278c-e933-4575-872b-3420353c400a"
Nov 01 03:41:59.152454 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.152273    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeReady"
Nov 01 03:41:59.152757 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.152551    2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 03:41:59.271923 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.271860    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6t6x6"]
Nov 01 03:41:59.274901 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.274882    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x52jq"]
Nov 01 03:41:59.275021 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.275003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6t6x6"
Nov 01 03:41:59.279219 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.279198    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"]
Nov 01 03:41:59.279308 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.279293    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-x52jq"
Nov 01 03:41:59.282710 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.282687    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wgcvj"]
Nov 01 03:41:59.282834 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.282815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:41:59.286227 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.286208    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"]
Nov 01 03:41:59.286305 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.286294    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wgcvj"
Nov 01 03:41:59.289811 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.289791    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:41:59.323735 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.323715    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-j945n"]
Nov 01 03:41:59.326945 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.326922    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"]
Nov 01 03:41:59.327050 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.327005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.330033 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.330001    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7dcc875c8f-dlqlr"]
Nov 01 03:41:59.330281 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.330249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:41:59.332416 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.332376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7vxk\" (UniqueName: \"kubernetes.io/projected/6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10-kube-api-access-b7vxk\") pod \"kas-connection-checker-685895c7c9-6t6x6\" (UID: \"6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10\") " pod="kube-system/kas-connection-checker-685895c7c9-6t6x6"
Nov 01 03:41:59.333268 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.333251    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md"]
Nov 01 03:41:59.333393 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.333377    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.336271 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.336254    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"]
Nov 01 03:41:59.336363 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.336350    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md"
Nov 01 03:41:59.339479 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.339460    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-74gj4"]
Nov 01 03:41:59.339569 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.339556    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:41:59.343386 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.343361    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8894dd498-k59gh"]
Nov 01 03:41:59.343602 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.343531    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.346858 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.346839    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-565fcbb6bc-wvnqs"]
Nov 01 03:41:59.346969 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.346953    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.350402 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.350385    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k"]
Nov 01 03:41:59.350526 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.350509    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.353263 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.353246    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"]
Nov 01 03:41:59.353350 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.353339    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k"
Nov 01 03:41:59.357619 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.357603    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7v4p5"]
Nov 01 03:41:59.357803 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.357782    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:41:59.360867 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.360848    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-46gkp"]
Nov 01 03:41:59.360960 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.360930    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.364044 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.364023    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6t6x6"]
Nov 01 03:41:59.364142 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.364113    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:41:59.375062 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.375035    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 03:41:59.375242 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.375153    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 03:41:59.375947 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.375925    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"]
Nov 01 03:41:59.376654 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.376641    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x52jq"]
Nov 01 03:41:59.377384 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.377363    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wgcvj"]
Nov 01 03:41:59.377875 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.377849    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 03:41:59.377953 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.377905    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 03:41:59.377953 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.377923    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j6wvf\""
Nov 01 03:41:59.378915 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.378210    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"]
Nov 01 03:41:59.379235 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.379222    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md"]
Nov 01 03:41:59.379975 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.379955    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7dcc875c8f-dlqlr"]
Nov 01 03:41:59.380926 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.380909    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-74gj4"]
Nov 01 03:41:59.381773 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.381743    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8894dd498-k59gh"]
Nov 01 03:41:59.383384 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.383151    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-p85rd\""
Nov 01 03:41:59.384674 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.384656    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-46gkp"]
Nov 01 03:41:59.387279 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.387259    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-565fcbb6bc-wvnqs"]
Nov 01 03:41:59.391162 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.391142    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"]
Nov 01 03:41:59.394111 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.394092    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-j945n"]
Nov 01 03:41:59.420385 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.420368    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k"]
Nov 01 03:41:59.421683 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.421666    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"]
Nov 01 03:41:59.430142 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.430125    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"]
Nov 01 03:41:59.431717 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.431695    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 03:41:59.431827 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.431736    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 03:41:59.431827 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.431739    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 03:41:59.431827 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.431739    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:41:59.431955 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.431826    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 03:41:59.431955 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.431848    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 03:41:59.431955 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.431855    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 03:41:59.431955 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.431901    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:41:59.432352 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.432338    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 03:41:59.433088 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433069    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-bound-sa-token\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.433168 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-bound-sa-token\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.433168 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a62e18-e602-4b06-ae13-49276b60f80c-tmp-dir\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.433254 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433171    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-trusted-ca\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.433254 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b42gr\" (UniqueName: \"kubernetes.io/projected/cd1787d7-cd00-470d-b48f-c2d71e566dd2-kube-api-access-b42gr\") pod \"kas-connection-checker-685895c7c9-x52jq\" (UID: \"cd1787d7-cd00-470d-b48f-c2d71e566dd2\") " pod="kube-system/kas-connection-checker-685895c7c9-x52jq"
Nov 01 03:41:59.433328 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4070d3e-397e-4202-abb8-ff360ea75ee2-serving-cert\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.433328 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-certificates\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.433395 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.433395 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433379    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0271bbb9-50cc-47e2-8681-20e6b2550469-config\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:41:59.433512 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vxk\" (UniqueName: \"kubernetes.io/projected/6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10-kube-api-access-b7vxk\") pod \"kas-connection-checker-685895c7c9-6t6x6\" (UID: \"6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10\") " pod="kube-system/kas-connection-checker-685895c7c9-6t6x6"
Nov 01 03:41:59.433512 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-installation-pull-secrets\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.433608 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-default-certificate\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.433660 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.433660 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-trusted-ca\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.433660 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6979e621-5326-4fff-8641-af89a3065735-ca-trust-extracted\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.433660 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-installation-pull-secrets\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.433809 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:41:59.433809 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-snapshots\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.433809 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-service-ca-bundle\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.433911 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433814    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:41:59.433911 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433840    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjqww\" (UniqueName: \"kubernetes.io/projected/bd4e00b4-e891-4be2-b941-1238d43e8805-kube-api-access-vjqww\") pod \"ingress-canary-46gkp\" (UID: \"bd4e00b4-e891-4be2-b941-1238d43e8805\") " pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:41:59.433911 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc4z5\" (UniqueName: \"kubernetes.io/projected/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-kube-api-access-zc4z5\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:41:59.434054 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-image-registry-private-configuration\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.434054 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.433984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-stats-auth\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.434054 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.434150 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434071    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxjcw\" (UniqueName: \"kubernetes.io/projected/f4070d3e-397e-4202-abb8-ff360ea75ee2-kube-api-access-gxjcw\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.434150 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.434150 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.434255 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0271bbb9-50cc-47e2-8681-20e6b2550469-serving-cert\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:41:59.434255 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434174    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-config\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.434255 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:41:59.434255 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mctgm\" (UniqueName: \"kubernetes.io/projected/91524329-4a22-4667-bdeb-fd72ef337912-kube-api-access-mctgm\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:41:59.434440 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmxpq\" (UniqueName: \"kubernetes.io/projected/f9f6a2a3-88f9-4d37-b144-371147dd6a6e-kube-api-access-hmxpq\") pod \"network-check-source-6f78d59f99-lfx6k\" (UID: \"f9f6a2a3-88f9-4d37-b144-371147dd6a6e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k"
Nov 01 03:41:59.434440 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:41:59.434440 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd4e00b4-e891-4be2-b941-1238d43e8805-cert\") pod \"ingress-canary-46gkp\" (UID: \"bd4e00b4-e891-4be2-b941-1238d43e8805\") " pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:41:59.434440 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.434440 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:41:59.434440 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7c51022-86cd-4cac-bee9-22a33fd53e22-ca-trust-extracted\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.434646 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-image-registry-private-configuration\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.434646 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434483    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:41:59.434646 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8lp\" (UniqueName: \"kubernetes.io/projected/19a62e18-e602-4b06-ae13-49276b60f80c-kube-api-access-2c8lp\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.434646 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434535    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmbq\" (UniqueName: \"kubernetes.io/projected/0271bbb9-50cc-47e2-8681-20e6b2550469-kube-api-access-rqmbq\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:41:59.434646 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-serving-cert\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.434646 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqvdm\" (UniqueName: \"kubernetes.io/projected/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-kube-api-access-lqvdm\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.434646 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434612    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19a62e18-e602-4b06-ae13-49276b60f80c-config-volume\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.434646 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434628    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqjhm\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-kube-api-access-kqjhm\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.434865 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pxlp\" (UniqueName: \"kubernetes.io/projected/89e71943-93a0-48b5-aa15-e060cc7b30f3-kube-api-access-5pxlp\") pod \"volume-data-source-validator-58c9f5f46d-5t7md\" (UID: \"89e71943-93a0-48b5-aa15-e060cc7b30f3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md"
Nov 01 03:41:59.434865 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfd98\" (UniqueName: \"kubernetes.io/projected/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-kube-api-access-mfd98\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:41:59.434865 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434752    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-tmp\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.434865 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:41:59.434865 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n9m2\" (UniqueName: \"kubernetes.io/projected/036c1722-9db5-406e-bc09-0aa6f215e1e0-kube-api-access-4n9m2\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.434865 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msg64\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-kube-api-access-msg64\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.435100 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-registry-certificates\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.435100 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:41:59.435100 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.434975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z644b\" (UniqueName: \"kubernetes.io/projected/cd24318e-36ed-4fbb-9742-e6e2a9486283-kube-api-access-z644b\") pod \"kas-connection-checker-685895c7c9-wgcvj\" (UID: \"cd24318e-36ed-4fbb-9742-e6e2a9486283\") " pod="kube-system/kas-connection-checker-685895c7c9-wgcvj"
Nov 01 03:41:59.435100 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.435023    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-trusted-ca\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.472896 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.472856    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 03:41:59.472896 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.472870    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 03:41:59.472896 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.472870    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:41:59.473088 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.472945    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 03:41:59.473088 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.473000    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 03:41:59.477924 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.477910    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 03:41:59.480887 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.480873    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 03:41:59.480950 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.480937    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 03:41:59.484108 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484094    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:41:59.484177 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484164    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 03:41:59.484348 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484337    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 03:41:59.484403 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484390    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 03:41:59.484460 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484419    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 03:41:59.484505 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484493    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 03:41:59.484600 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484581    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 03:41:59.484687 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484672    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 03:41:59.484743 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484687    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:41:59.484743 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484689    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 03:41:59.484743 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484710    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 03:41:59.484853 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484788    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 03:41:59.484853 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.484829    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 03:41:59.487259 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.487244    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 03:41:59.487438 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.487415    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 03:41:59.490687 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.490567    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 03:41:59.490687 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.490650    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 03:41:59.490687 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.490658    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 03:41:59.490894 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.490691    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 03:41:59.490894 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.490712    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 03:41:59.490894 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.490738    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 03:41:59.490894 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.490744    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 03:41:59.490894 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.490747    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 03:41:59.490894 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.490741    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 03:41:59.490894 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.490756    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 03:41:59.490894 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.490780    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j4nxv\""
Nov 01 03:41:59.491788 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.491773    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 03:41:59.491869 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.491815    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 03:41:59.491921 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.491876    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g6x5b\""
Nov 01 03:41:59.492245 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.492228    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7v4p5"]
Nov 01 03:41:59.496935 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.496922    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-75k7x\""
Nov 01 03:41:59.497144 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.497129    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2mnqj\""
Nov 01 03:41:59.497232 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.497132    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jx8x8\""
Nov 01 03:41:59.497232 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.497174    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jgbfn\""
Nov 01 03:41:59.497232 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.497187    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 03:41:59.497232 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.497196    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7gg6p\""
Nov 01 03:41:59.497403 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.497236    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-wjdgr\""
Nov 01 03:41:59.501240 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.501218    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7vxk\" (UniqueName: \"kubernetes.io/projected/6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10-kube-api-access-b7vxk\") pod \"kas-connection-checker-685895c7c9-6t6x6\" (UID: \"6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10\") " pod="kube-system/kas-connection-checker-685895c7c9-6t6x6"
Nov 01 03:41:59.536062 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-tmp\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.536062 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:41:59.536062 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9m2\" (UniqueName: \"kubernetes.io/projected/036c1722-9db5-406e-bc09-0aa6f215e1e0-kube-api-access-4n9m2\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.536241 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msg64\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-kube-api-access-msg64\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.536241 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.536182    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:41:59.536241 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-registry-certificates\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.536390 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.536247    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls podName:91524329-4a22-4667-bdeb-fd72ef337912 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:00.036228007 +0000 UTC m=+32.739459850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zvcfn" (UID: "91524329-4a22-4667-bdeb-fd72ef337912") : secret "samples-operator-tls" not found
Nov 01 03:41:59.536390 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:41:59.536390 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536313    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z644b\" (UniqueName: \"kubernetes.io/projected/cd24318e-36ed-4fbb-9742-e6e2a9486283-kube-api-access-z644b\") pod \"kas-connection-checker-685895c7c9-wgcvj\" (UID: \"cd24318e-36ed-4fbb-9742-e6e2a9486283\") " pod="kube-system/kas-connection-checker-685895c7c9-wgcvj"
Nov 01 03:41:59.536390 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-trusted-ca\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.536390 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536368    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-bound-sa-token\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.536658 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-bound-sa-token\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.536658 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536408    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-tmp\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.536658 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536422    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a62e18-e602-4b06-ae13-49276b60f80c-tmp-dir\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.536658 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-trusted-ca\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.536843 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42gr\" (UniqueName: \"kubernetes.io/projected/cd1787d7-cd00-470d-b48f-c2d71e566dd2-kube-api-access-b42gr\") pod \"kas-connection-checker-685895c7c9-x52jq\" (UID: \"cd1787d7-cd00-470d-b48f-c2d71e566dd2\") " pod="kube-system/kas-connection-checker-685895c7c9-x52jq"
Nov 01 03:41:59.536843 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4070d3e-397e-4202-abb8-ff360ea75ee2-serving-cert\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.536843 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-certificates\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.536843 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.536843 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536804    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-registry-certificates\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.536843 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536824    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0271bbb9-50cc-47e2-8681-20e6b2550469-config\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:41:59.537118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-installation-pull-secrets\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.537118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-default-certificate\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.537118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.537118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537004    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-trusted-ca\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.537118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6979e621-5326-4fff-8641-af89a3065735-ca-trust-extracted\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.537118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537053    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-installation-pull-secrets\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.537118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537087    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:41:59.537118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537099    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:41:59.537118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-snapshots\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.537558 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537362    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-trusted-ca\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.537558 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537410    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0271bbb9-50cc-47e2-8681-20e6b2550469-config\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:41:59.537558 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537456    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-trusted-ca\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.537558 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.537492    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:00.037475864 +0000 UTC m=+32.740707708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : configmap references non-existent config key: service-ca.crt
Nov 01 03:41:59.537782 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537602    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-certificates\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.537782 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537621    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-snapshots\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.537782 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.537715    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:41:59.537782 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537731    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6979e621-5326-4fff-8641-af89a3065735-ca-trust-extracted\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.537782 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.537771    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls podName:d50a0b39-914c-4b62-9b0b-4c6ab787f9d7 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:00.037749036 +0000 UTC m=+32.740980879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pknwh" (UID: "d50a0b39-914c-4b62-9b0b-4c6ab787f9d7") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:41:59.538028 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.536773    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a62e18-e602-4b06-ae13-49276b60f80c-tmp-dir\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.538028 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-service-ca-bundle\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.538028 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:41:59.538028 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537919    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.538028 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjqww\" (UniqueName: \"kubernetes.io/projected/bd4e00b4-e891-4be2-b941-1238d43e8805-kube-api-access-vjqww\") pod \"ingress-canary-46gkp\" (UID: \"bd4e00b4-e891-4be2-b941-1238d43e8805\") " pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:41:59.538028 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.537970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4z5\" (UniqueName: \"kubernetes.io/projected/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-kube-api-access-zc4z5\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:41:59.538028 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-image-registry-private-configuration\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-stats-auth\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxjcw\" (UniqueName: \"kubernetes.io/projected/f4070d3e-397e-4202-abb8-ff360ea75ee2-kube-api-access-gxjcw\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0271bbb9-50cc-47e2-8681-20e6b2550469-serving-cert\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-config\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mctgm\" (UniqueName: \"kubernetes.io/projected/91524329-4a22-4667-bdeb-fd72ef337912-kube-api-access-mctgm\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxpq\" (UniqueName: \"kubernetes.io/projected/f9f6a2a3-88f9-4d37-b144-371147dd6a6e-kube-api-access-hmxpq\") pod \"network-check-source-6f78d59f99-lfx6k\" (UID: \"f9f6a2a3-88f9-4d37-b144-371147dd6a6e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:41:59.538349 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd4e00b4-e891-4be2-b941-1238d43e8805-cert\") pod \"ingress-canary-46gkp\" (UID: \"bd4e00b4-e891-4be2-b941-1238d43e8805\") " pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538422    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7c51022-86cd-4cac-bee9-22a33fd53e22-ca-trust-extracted\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-image-registry-private-configuration\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538528    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8lp\" (UniqueName: \"kubernetes.io/projected/19a62e18-e602-4b06-ae13-49276b60f80c-kube-api-access-2c8lp\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmbq\" (UniqueName: \"kubernetes.io/projected/0271bbb9-50cc-47e2-8681-20e6b2550469-kube-api-access-rqmbq\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-serving-cert\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqvdm\" (UniqueName: \"kubernetes.io/projected/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-kube-api-access-lqvdm\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538673    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19a62e18-e602-4b06-ae13-49276b60f80c-config-volume\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqjhm\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-kube-api-access-kqjhm\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pxlp\" (UniqueName: \"kubernetes.io/projected/89e71943-93a0-48b5-aa15-e060cc7b30f3-kube-api-access-5pxlp\") pod \"volume-data-source-validator-58c9f5f46d-5t7md\" (UID: \"89e71943-93a0-48b5-aa15-e060cc7b30f3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md"
Nov 01 03:41:59.538931 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.538756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfd98\" (UniqueName: \"kubernetes.io/projected/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-kube-api-access-mfd98\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:41:59.539525 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.539108    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-trusted-ca\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.539591 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.539559    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4070d3e-397e-4202-abb8-ff360ea75ee2-serving-cert\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.539671 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.539650    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-service-ca-bundle\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.539756 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.539742    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:41:59.539831 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.539821    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:00.039806159 +0000 UTC m=+32.743038003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : secret "router-metrics-certs-default" not found
Nov 01 03:41:59.539896 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.539878    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-installation-pull-secrets\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.539986 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.539971    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:41:59.540043 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.540030    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert podName:5cf6ba1f-8ecb-46fd-a2b9-db4319378237 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:00.040015701 +0000 UTC m=+32.743247546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bnwm" (UID: "5cf6ba1f-8ecb-46fd-a2b9-db4319378237") : secret "networking-console-plugin-cert" not found
Nov 01 03:41:59.540106 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.540082    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:41:59.540164 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.540126    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-default-certificate\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.540276 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.540254    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-installation-pull-secrets\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.540413 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.540395    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:41:59.540754 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.540736    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:41:59.540754 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.540755    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565fcbb6bc-wvnqs: secret "image-registry-tls" not found
Nov 01 03:41:59.540867 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.540806    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls podName:c7c51022-86cd-4cac-bee9-22a33fd53e22 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:00.04079244 +0000 UTC m=+32.744024284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls") pod "image-registry-565fcbb6bc-wvnqs" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22") : secret "image-registry-tls" not found
Nov 01 03:41:59.540936 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.540880    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-config\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.542185 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.541202    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls podName:19a62e18-e602-4b06-ae13-49276b60f80c nodeName:}" failed. No retries permitted until 2026-11-01 03:42:00.041184264 +0000 UTC m=+32.744416099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls") pod "dns-default-7v4p5" (UID: "19a62e18-e602-4b06-ae13-49276b60f80c") : secret "dns-default-metrics-tls" not found
Nov 01 03:41:59.542185 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.541527    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:41:59.542185 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.541587    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd4e00b4-e891-4be2-b941-1238d43e8805-cert\") pod \"ingress-canary-46gkp\" (UID: \"bd4e00b4-e891-4be2-b941-1238d43e8805\") " pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:41:59.542185 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.541743    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7c51022-86cd-4cac-bee9-22a33fd53e22-ca-trust-extracted\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.542445 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.542326    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-image-registry-private-configuration\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.542445 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.542420    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:41:59.542545 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.542451    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8894dd498-k59gh: secret "image-registry-tls" not found
Nov 01 03:41:59.542545 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:41:59.542492    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls podName:6979e621-5326-4fff-8641-af89a3065735 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:00.042477729 +0000 UTC m=+32.745709569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls") pod "image-registry-8894dd498-k59gh" (UID: "6979e621-5326-4fff-8641-af89a3065735") : secret "image-registry-tls" not found
Nov 01 03:41:59.542545 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.542505    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:41:59.542700 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.542655    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19a62e18-e602-4b06-ae13-49276b60f80c-config-volume\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.543002 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.542976    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:41:59.543070 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.543003    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0271bbb9-50cc-47e2-8681-20e6b2550469-serving-cert\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:41:59.543356 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.543337    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-stats-auth\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.543833 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.543812    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-image-registry-private-configuration\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.544129 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.544113    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-serving-cert\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.589178 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.589156    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6t6x6"
Nov 01 03:41:59.600377 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.600083    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-bound-sa-token\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.600377 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.600220    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42gr\" (UniqueName: \"kubernetes.io/projected/cd1787d7-cd00-470d-b48f-c2d71e566dd2-kube-api-access-b42gr\") pod \"kas-connection-checker-685895c7c9-x52jq\" (UID: \"cd1787d7-cd00-470d-b48f-c2d71e566dd2\") " pod="kube-system/kas-connection-checker-685895c7c9-x52jq"
Nov 01 03:41:59.600541 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.600377    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z644b\" (UniqueName: \"kubernetes.io/projected/cd24318e-36ed-4fbb-9742-e6e2a9486283-kube-api-access-z644b\") pod \"kas-connection-checker-685895c7c9-wgcvj\" (UID: \"cd24318e-36ed-4fbb-9742-e6e2a9486283\") " pod="kube-system/kas-connection-checker-685895c7c9-wgcvj"
Nov 01 03:41:59.600655 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.600632    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msg64\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-kube-api-access-msg64\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:41:59.600732 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.600717    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9m2\" (UniqueName: \"kubernetes.io/projected/036c1722-9db5-406e-bc09-0aa6f215e1e0-kube-api-access-4n9m2\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:41:59.600994 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.600967    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-bound-sa-token\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.602239 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.602219    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mctgm\" (UniqueName: \"kubernetes.io/projected/91524329-4a22-4667-bdeb-fd72ef337912-kube-api-access-mctgm\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:41:59.602519 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.602484    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfd98\" (UniqueName: \"kubernetes.io/projected/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-kube-api-access-mfd98\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:41:59.603122 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.603103    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4z5\" (UniqueName: \"kubernetes.io/projected/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-kube-api-access-zc4z5\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:41:59.604516 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.604491    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmbq\" (UniqueName: \"kubernetes.io/projected/0271bbb9-50cc-47e2-8681-20e6b2550469-kube-api-access-rqmbq\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:41:59.604800 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.604779    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxpq\" (UniqueName: \"kubernetes.io/projected/f9f6a2a3-88f9-4d37-b144-371147dd6a6e-kube-api-access-hmxpq\") pod \"network-check-source-6f78d59f99-lfx6k\" (UID: \"f9f6a2a3-88f9-4d37-b144-371147dd6a6e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k"
Nov 01 03:41:59.605691 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.605670    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wgcvj"
Nov 01 03:41:59.606243 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.606223    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjqww\" (UniqueName: \"kubernetes.io/projected/bd4e00b4-e891-4be2-b941-1238d43e8805-kube-api-access-vjqww\") pod \"ingress-canary-46gkp\" (UID: \"bd4e00b4-e891-4be2-b941-1238d43e8805\") " pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:41:59.607633 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.607417    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8lp\" (UniqueName: \"kubernetes.io/projected/19a62e18-e602-4b06-ae13-49276b60f80c-kube-api-access-2c8lp\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:41:59.607633 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.607590    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqjhm\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-kube-api-access-kqjhm\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:41:59.607813 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.607794    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pxlp\" (UniqueName: \"kubernetes.io/projected/89e71943-93a0-48b5-aa15-e060cc7b30f3-kube-api-access-5pxlp\") pod \"volume-data-source-validator-58c9f5f46d-5t7md\" (UID: \"89e71943-93a0-48b5-aa15-e060cc7b30f3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md"
Nov 01 03:41:59.609242 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.609195    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqvdm\" (UniqueName: \"kubernetes.io/projected/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-kube-api-access-lqvdm\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.609455 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.609416    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxjcw\" (UniqueName: \"kubernetes.io/projected/f4070d3e-397e-4202-abb8-ff360ea75ee2-kube-api-access-gxjcw\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.642136 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.642101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:41:59.650525 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.650495    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:41:59.669788 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.669221    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md"
Nov 01 03:41:59.679353 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.679326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:41:59.703178 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.700810    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k"
Nov 01 03:41:59.707456 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.706117    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:41:59.716111 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.715919    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:41:59.750806 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.750585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:41:59.750806 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.750644    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:41:59.750806 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.750644    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:41:59.805691 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.805666    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 03:41:59.805815 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.805770    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v5dfc\""
Nov 01 03:41:59.805879 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.805837    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-69w79\""
Nov 01 03:41:59.805879 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.805863    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 03:41:59.805984 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.805921    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jzxr2\""
Nov 01 03:41:59.895105 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.895075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-x52jq"
Nov 01 03:41:59.986854 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.986826    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wgcvj"]
Nov 01 03:41:59.989922 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:41:59.989897    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6t6x6"]
Nov 01 03:41:59.990972 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:59.990884    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd24318e_36ed_4fbb_9742_e6e2a9486283.slice/crio-7969e4f0dc467b9847fbb4383dda48e41a9d3b0bbbb3128df6cdbb1beed0063f WatchSource:0}: Error finding container 7969e4f0dc467b9847fbb4383dda48e41a9d3b0bbbb3128df6cdbb1beed0063f: Status 404 returned error can't find the container with id 7969e4f0dc467b9847fbb4383dda48e41a9d3b0bbbb3128df6cdbb1beed0063f
Nov 01 03:41:59.993216 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:41:59.993184    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4a6bb3_f059_4fcf_9d3f_af5667cf7f10.slice/crio-599b58367432f5e13abaea4a8c58ff70eafe140d71e4db85b9d8b343f38ff892 WatchSource:0}: Error finding container 599b58367432f5e13abaea4a8c58ff70eafe140d71e4db85b9d8b343f38ff892: Status 404 returned error can't find the container with id 599b58367432f5e13abaea4a8c58ff70eafe140d71e4db85b9d8b343f38ff892
Nov 01 03:42:00.046944 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.046866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:00.046944 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.046931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:00.047124 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.046981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:00.047124 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047027    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:42:00.047124 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047103    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls podName:91524329-4a22-4667-bdeb-fd72ef337912 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:01.047082693 +0000 UTC m=+33.750314523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zvcfn" (UID: "91524329-4a22-4667-bdeb-fd72ef337912") : secret "samples-operator-tls" not found
Nov 01 03:42:00.047124 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.047035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:00.047282 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.047152    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:00.047282 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.047192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:00.047282 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.047221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:42:00.047282 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047251    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:42:00.047282 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.047263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:00.047457 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047302    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls podName:19a62e18-e602-4b06-ae13-49276b60f80c nodeName:}" failed. No retries permitted until 2026-11-01 03:42:01.047287494 +0000 UTC m=+33.750519335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls") pod "dns-default-7v4p5" (UID: "19a62e18-e602-4b06-ae13-49276b60f80c") : secret "dns-default-metrics-tls" not found
Nov 01 03:42:00.047457 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047375    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:00.047457 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047380    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:01.047369376 +0000 UTC m=+33.750601226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : configmap references non-existent config key: service-ca.crt
Nov 01 03:42:00.047457 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047387    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565fcbb6bc-wvnqs: secret "image-registry-tls" not found
Nov 01 03:42:00.047457 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047447    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls podName:c7c51022-86cd-4cac-bee9-22a33fd53e22 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:01.047411912 +0000 UTC m=+33.750643755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls") pod "image-registry-565fcbb6bc-wvnqs" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22") : secret "image-registry-tls" not found
Nov 01 03:42:00.047693 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047461    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:00.047693 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047495    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls podName:d50a0b39-914c-4b62-9b0b-4c6ab787f9d7 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:01.047485804 +0000 UTC m=+33.750717662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pknwh" (UID: "d50a0b39-914c-4b62-9b0b-4c6ab787f9d7") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:00.047693 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047505    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:00.047693 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047514    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8894dd498-k59gh: secret "image-registry-tls" not found
Nov 01 03:42:00.047693 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047543    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls podName:6979e621-5326-4fff-8641-af89a3065735 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:01.047533838 +0000 UTC m=+33.750765700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls") pod "image-registry-8894dd498-k59gh" (UID: "6979e621-5326-4fff-8641-af89a3065735") : secret "image-registry-tls" not found
Nov 01 03:42:00.047693 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047546    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:42:00.047693 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047575    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:01.047566206 +0000 UTC m=+33.750798050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : secret "router-metrics-certs-default" not found
Nov 01 03:42:00.047693 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047596    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:42:00.047693 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:00.047625    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert podName:5cf6ba1f-8ecb-46fd-a2b9-db4319378237 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:01.047616332 +0000 UTC m=+33.750848186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bnwm" (UID: "5cf6ba1f-8ecb-46fd-a2b9-db4319378237") : secret "networking-console-plugin-cert" not found
Nov 01 03:42:00.052825 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.051492    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-j945n"]
Nov 01 03:42:00.052825 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.052466    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"]
Nov 01 03:42:00.086247 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.086222    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-74gj4"]
Nov 01 03:42:00.091110 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:42:00.091066    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4070d3e_397e_4202_abb8_ff360ea75ee2.slice/crio-61008947d7850e6869cb38b5a0c920216d88e89cdf30c870a696cdce60662848 WatchSource:0}: Error finding container 61008947d7850e6869cb38b5a0c920216d88e89cdf30c870a696cdce60662848: Status 404 returned error can't find the container with id 61008947d7850e6869cb38b5a0c920216d88e89cdf30c870a696cdce60662848
Nov 01 03:42:00.127548 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.127524    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md"]
Nov 01 03:42:00.131029 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:42:00.131008    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e71943_93a0_48b5_aa15_e060cc7b30f3.slice/crio-b0f5216e5bbf8b41fc984fc29d3fca32dd91c056efedd73158eb328680f5344a WatchSource:0}: Error finding container b0f5216e5bbf8b41fc984fc29d3fca32dd91c056efedd73158eb328680f5344a: Status 404 returned error can't find the container with id b0f5216e5bbf8b41fc984fc29d3fca32dd91c056efedd73158eb328680f5344a
Nov 01 03:42:00.169652 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.169625    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k"]
Nov 01 03:42:00.172893 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:42:00.172867    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f6a2a3_88f9_4d37_b144_371147dd6a6e.slice/crio-ea73befeff421f14d4c19072786a918dbffe0de1b2fa777b588de37298efbbba WatchSource:0}: Error finding container ea73befeff421f14d4c19072786a918dbffe0de1b2fa777b588de37298efbbba: Status 404 returned error can't find the container with id ea73befeff421f14d4c19072786a918dbffe0de1b2fa777b588de37298efbbba
Nov 01 03:42:00.190715 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.190663    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"]
Nov 01 03:42:00.193910 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:42:00.193884    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0271bbb9_50cc_47e2_8681_20e6b2550469.slice/crio-8eb74a593784fbcf0b3f74c2d761707f1a43a018745a6a759191727a4b627f98 WatchSource:0}: Error finding container 8eb74a593784fbcf0b3f74c2d761707f1a43a018745a6a759191727a4b627f98: Status 404 returned error can't find the container with id 8eb74a593784fbcf0b3f74c2d761707f1a43a018745a6a759191727a4b627f98
Nov 01 03:42:00.277264 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.277229    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x52jq"]
Nov 01 03:42:00.280941 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:42:00.280914    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1787d7_cd00_470d_b48f_c2d71e566dd2.slice/crio-45e31a093acba39081e9a7a984525dbc3f7e09bf532dfb053c963b0872613f55 WatchSource:0}: Error finding container 45e31a093acba39081e9a7a984525dbc3f7e09bf532dfb053c963b0872613f55: Status 404 returned error can't find the container with id 45e31a093acba39081e9a7a984525dbc3f7e09bf532dfb053c963b0872613f55
Nov 01 03:42:00.308173 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.308106    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-46gkp"]
Nov 01 03:42:00.311881 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:42:00.311853    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4e00b4_e891_4be2_b941_1238d43e8805.slice/crio-f21616762aa00d7dc8274121d13f5eb93b48968cc729caf759e7348edda9b227 WatchSource:0}: Error finding container f21616762aa00d7dc8274121d13f5eb93b48968cc729caf759e7348edda9b227: Status 404 returned error can't find the container with id f21616762aa00d7dc8274121d13f5eb93b48968cc729caf759e7348edda9b227
Nov 01 03:42:00.927149 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.927110    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-x52jq" event={"ID":"cd1787d7-cd00-470d-b48f-c2d71e566dd2","Type":"ContainerStarted","Data":"5ea3893f18394009929bbbc77e5643b3b1e41e419faedb750f7ac00c313c5438"}
Nov 01 03:42:00.927351 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.927173    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-x52jq" event={"ID":"cd1787d7-cd00-470d-b48f-c2d71e566dd2","Type":"ContainerStarted","Data":"45e31a093acba39081e9a7a984525dbc3f7e09bf532dfb053c963b0872613f55"}
Nov 01 03:42:00.928736 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.928707    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k" event={"ID":"f9f6a2a3-88f9-4d37-b144-371147dd6a6e","Type":"ContainerStarted","Data":"ea73befeff421f14d4c19072786a918dbffe0de1b2fa777b588de37298efbbba"}
Nov 01 03:42:00.929839 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.929810    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-74gj4" event={"ID":"f4070d3e-397e-4202-abb8-ff360ea75ee2","Type":"ContainerStarted","Data":"61008947d7850e6869cb38b5a0c920216d88e89cdf30c870a696cdce60662848"}
Nov 01 03:42:00.930996 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.930972    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n" event={"ID":"b5c8ccd8-df33-4df9-8311-5a163e8e31b3","Type":"ContainerStarted","Data":"b97a30eaa2bd4a1e67bfb448f070dd06f8e4387d1643f701058178d9fa617db3"}
Nov 01 03:42:00.932754 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.932733    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wgcvj" event={"ID":"cd24318e-36ed-4fbb-9742-e6e2a9486283","Type":"ContainerStarted","Data":"ba399560d6bc423ad39a791b583fe7967a45b9d0d2eb1950d2a9f7fd60a5dcfe"}
Nov 01 03:42:00.932852 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.932775    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wgcvj" event={"ID":"cd24318e-36ed-4fbb-9742-e6e2a9486283","Type":"ContainerStarted","Data":"7969e4f0dc467b9847fbb4383dda48e41a9d3b0bbbb3128df6cdbb1beed0063f"}
Nov 01 03:42:00.933810 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.933789    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98" event={"ID":"0271bbb9-50cc-47e2-8681-20e6b2550469","Type":"ContainerStarted","Data":"8eb74a593784fbcf0b3f74c2d761707f1a43a018745a6a759191727a4b627f98"}
Nov 01 03:42:00.934727 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.934709    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md" event={"ID":"89e71943-93a0-48b5-aa15-e060cc7b30f3","Type":"ContainerStarted","Data":"b0f5216e5bbf8b41fc984fc29d3fca32dd91c056efedd73158eb328680f5344a"}
Nov 01 03:42:00.935682 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.935663    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx" event={"ID":"3cd84ba1-3255-444e-b2e0-2a3d4ac25988","Type":"ContainerStarted","Data":"9a302cc5ca4e5b357243b484318ba248f37ee25402ccb51fe546813105e745e7"}
Nov 01 03:42:00.936765 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.936747    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-46gkp" event={"ID":"bd4e00b4-e891-4be2-b941-1238d43e8805","Type":"ContainerStarted","Data":"f21616762aa00d7dc8274121d13f5eb93b48968cc729caf759e7348edda9b227"}
Nov 01 03:42:00.937950 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.937929    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6t6x6" event={"ID":"6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10","Type":"ContainerStarted","Data":"fc6fbfb0acf99ca6b5595b75091766007bf07961c6dc00bfc954c540b87f015d"}
Nov 01 03:42:00.938019 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:00.937966    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6t6x6" event={"ID":"6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10","Type":"ContainerStarted","Data":"599b58367432f5e13abaea4a8c58ff70eafe140d71e4db85b9d8b343f38ff892"}
Nov 01 03:42:01.054950 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.054921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:01.055089 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.054970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:01.055089 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.054988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:01.055089 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055062    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:01.055089 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055074    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8894dd498-k59gh: secret "image-registry-tls" not found
Nov 01 03:42:01.055089 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.055082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055114    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls podName:6979e621-5326-4fff-8641-af89a3065735 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:03.055103347 +0000 UTC m=+35.758335173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls") pod "image-registry-8894dd498-k59gh" (UID: "6979e621-5326-4fff-8641-af89a3065735") : secret "image-registry-tls" not found
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.055128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055135    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055060    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.055155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055175    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:03.055162866 +0000 UTC m=+35.758394696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : secret "router-metrics-certs-default" not found
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055060    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055185    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055195    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls podName:19a62e18-e602-4b06-ae13-49276b60f80c nodeName:}" failed. No retries permitted until 2026-11-01 03:42:03.055189434 +0000 UTC m=+35.758421261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls") pod "dns-default-7v4p5" (UID: "19a62e18-e602-4b06-ae13-49276b60f80c") : secret "dns-default-metrics-tls" not found
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055204    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055213    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565fcbb6bc-wvnqs: secret "image-registry-tls" not found
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055221    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls podName:d50a0b39-914c-4b62-9b0b-4c6ab787f9d7 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:03.055203111 +0000 UTC m=+35.758434950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pknwh" (UID: "d50a0b39-914c-4b62-9b0b-4c6ab787f9d7") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055255    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert podName:5cf6ba1f-8ecb-46fd-a2b9-db4319378237 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:03.055244754 +0000 UTC m=+35.758476586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bnwm" (UID: "5cf6ba1f-8ecb-46fd-a2b9-db4319378237") : secret "networking-console-plugin-cert" not found
Nov 01 03:42:01.055330 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055275    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls podName:c7c51022-86cd-4cac-bee9-22a33fd53e22 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:03.055265916 +0000 UTC m=+35.758497749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls") pod "image-registry-565fcbb6bc-wvnqs" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22") : secret "image-registry-tls" not found
Nov 01 03:42:01.056027 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.055351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:01.056027 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055484    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:42:01.056027 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055528    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls podName:91524329-4a22-4667-bdeb-fd72ef337912 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:03.055517252 +0000 UTC m=+35.758749099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zvcfn" (UID: "91524329-4a22-4667-bdeb-fd72ef337912") : secret "samples-operator-tls" not found
Nov 01 03:42:01.056027 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.055480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:01.056027 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.055557    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:03.055545573 +0000 UTC m=+35.758777431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : configmap references non-existent config key: service-ca.crt
Nov 01 03:42:01.563070 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.563008    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6t6x6" podStartSLOduration=423.562990125 podStartE2EDuration="7m3.562990125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:01.561792443 +0000 UTC m=+34.265024321" watchObservedRunningTime="2026-11-01 03:42:01.562990125 +0000 UTC m=+34.266221977"
Nov 01 03:42:01.661282 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.661240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:01.661445 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.661407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:42:01.661718 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.661696    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:42:01.661806 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:01.661788    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs podName:b496de82-bd77-42c5-8a29-e38514f99c5d nodeName:}" failed. No retries permitted until 2026-11-01 03:42:33.661766442 +0000 UTC m=+66.364998271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs") pod "network-metrics-daemon-vqrc7" (UID: "b496de82-bd77-42c5-8a29-e38514f99c5d") : secret "metrics-daemon-secret" not found
Nov 01 03:42:01.664575 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.664527    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:42:01.691020 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.690955    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-x52jq" podStartSLOduration=423.690937711 podStartE2EDuration="7m3.690937711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:01.690358348 +0000 UTC m=+34.393590198" watchObservedRunningTime="2026-11-01 03:42:01.690937711 +0000 UTC m=+34.394169561"
Nov 01 03:42:01.761946 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.761911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcbx\" (UniqueName: \"kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx\") pod \"network-check-target-hbgqw\" (UID: \"42ba278c-e933-4575-872b-3420353c400a\") " pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:42:01.764811 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.764782    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcbx\" (UniqueName: \"kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx\") pod \"network-check-target-hbgqw\" (UID: \"42ba278c-e933-4575-872b-3420353c400a\") " pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:42:01.810687 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.810413    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wgcvj" podStartSLOduration=423.810390646 podStartE2EDuration="7m3.810390646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:01.809496928 +0000 UTC m=+34.512759207" watchObservedRunningTime="2026-11-01 03:42:01.810390646 +0000 UTC m=+34.513622496"
Nov 01 03:42:01.896914 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.896839    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:42:01.914644 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:01.914617    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:42:02.282583 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:02.282550    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zdw9t"]
Nov 01 03:42:02.286673 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:42:02.286643    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f80953_074f_455a_9c88_6b3154926050.slice/crio-821b9ce2527df5ee5e49cb7f6f27a60a82b469a8aa7c2c188e4ee93c8f28673d WatchSource:0}: Error finding container 821b9ce2527df5ee5e49cb7f6f27a60a82b469a8aa7c2c188e4ee93c8f28673d: Status 404 returned error can't find the container with id 821b9ce2527df5ee5e49cb7f6f27a60a82b469a8aa7c2c188e4ee93c8f28673d
Nov 01 03:42:02.340181 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:02.340108    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hbgqw"]
Nov 01 03:42:02.344349 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:42:02.344295    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ba278c_e933_4575_872b_3420353c400a.slice/crio-0e2eeb1496827ebfe8a3d759fbde6b886d3bfc63aa5a41eb4c005da35e24511c WatchSource:0}: Error finding container 0e2eeb1496827ebfe8a3d759fbde6b886d3bfc63aa5a41eb4c005da35e24511c: Status 404 returned error can't find the container with id 0e2eeb1496827ebfe8a3d759fbde6b886d3bfc63aa5a41eb4c005da35e24511c
Nov 01 03:42:02.946755 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:02.946712    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hbgqw" event={"ID":"42ba278c-e933-4575-872b-3420353c400a","Type":"ContainerStarted","Data":"0e2eeb1496827ebfe8a3d759fbde6b886d3bfc63aa5a41eb4c005da35e24511c"}
Nov 01 03:42:02.948946 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:02.948885    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zdw9t" event={"ID":"b3f80953-074f-455a-9c88-6b3154926050","Type":"ContainerStarted","Data":"821b9ce2527df5ee5e49cb7f6f27a60a82b469a8aa7c2c188e4ee93c8f28673d"}
Nov 01 03:42:03.077775 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:03.077733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:03.077946 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:03.077812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:03.077946 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:03.077837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:03.077946 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:03.077865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:03.077946 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.077888    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:03.077946 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.077933    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:42:03.077946 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.077950    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls podName:d50a0b39-914c-4b62-9b0b-4c6ab787f9d7 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:07.077934851 +0000 UTC m=+39.781166679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pknwh" (UID: "d50a0b39-914c-4b62-9b0b-4c6ab787f9d7") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:03.078211 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.077956    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:42:03.078211 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.077994    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls podName:19a62e18-e602-4b06-ae13-49276b60f80c nodeName:}" failed. No retries permitted until 2026-11-01 03:42:07.077977413 +0000 UTC m=+39.781209243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls") pod "dns-default-7v4p5" (UID: "19a62e18-e602-4b06-ae13-49276b60f80c") : secret "dns-default-metrics-tls" not found
Nov 01 03:42:03.078211 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078013    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert podName:5cf6ba1f-8ecb-46fd-a2b9-db4319378237 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:07.078003232 +0000 UTC m=+39.781235065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bnwm" (UID: "5cf6ba1f-8ecb-46fd-a2b9-db4319378237") : secret "networking-console-plugin-cert" not found
Nov 01 03:42:03.078211 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078048    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:03.078211 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078059    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8894dd498-k59gh: secret "image-registry-tls" not found
Nov 01 03:42:03.078211 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078076    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:42:03.078211 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078094    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls podName:6979e621-5326-4fff-8641-af89a3065735 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:07.078080735 +0000 UTC m=+39.781312565 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls") pod "image-registry-8894dd498-k59gh" (UID: "6979e621-5326-4fff-8641-af89a3065735") : secret "image-registry-tls" not found
Nov 01 03:42:03.078211 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:03.077890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:42:03.078211 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078111    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:07.078100868 +0000 UTC m=+39.781332703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : secret "router-metrics-certs-default" not found
Nov 01 03:42:03.078211 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:03.078163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:03.078524 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:03.078226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:03.078524 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078310    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:03.078524 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078317    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:42:03.078524 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:03.078336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:03.078524 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078349    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls podName:91524329-4a22-4667-bdeb-fd72ef337912 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:07.078338884 +0000 UTC m=+39.781570713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zvcfn" (UID: "91524329-4a22-4667-bdeb-fd72ef337912") : secret "samples-operator-tls" not found
Nov 01 03:42:03.078524 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078389    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:07.078380623 +0000 UTC m=+39.781612450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : configmap references non-existent config key: service-ca.crt
Nov 01 03:42:03.078524 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078320    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565fcbb6bc-wvnqs: secret "image-registry-tls" not found
Nov 01 03:42:03.078524 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:03.078414    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls podName:c7c51022-86cd-4cac-bee9-22a33fd53e22 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:07.078407084 +0000 UTC m=+39.781638910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls") pod "image-registry-565fcbb6bc-wvnqs" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22") : secret "image-registry-tls" not found
Nov 01 03:42:03.957470 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:03.956775    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md" event={"ID":"89e71943-93a0-48b5-aa15-e060cc7b30f3","Type":"ContainerStarted","Data":"24ed264caef195d62c64298490ad7f4f2d60b6fb59c46b460f804bc42df338a5"}
Nov 01 03:42:04.078790 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:04.078733    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md" podStartSLOduration=407.75351887 podStartE2EDuration="6m51.07871901s" totalImagesPullingTime="3.32520014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:04.077418252 +0000 UTC m=+36.780650104" watchObservedRunningTime="2026-11-01 03:42:04.07871901 +0000 UTC m=+36.781950859"
Nov 01 03:42:04.962631 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:04.962588    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98" event={"ID":"0271bbb9-50cc-47e2-8681-20e6b2550469","Type":"ContainerStarted","Data":"76a16282a6954301d080e5d0b1d8462ce1f7468cc247f1a4295e9ac66d31c5e4"}
Nov 01 03:42:05.084138 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:05.084076    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98" podStartSLOduration=414.909974489 podStartE2EDuration="6m59.084055951s" totalImagesPullingTime="4.174081462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:05.082922167 +0000 UTC m=+37.786154018" watchObservedRunningTime="2026-11-01 03:42:05.084055951 +0000 UTC m=+37.787287802"
Nov 01 03:42:05.969063 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:05.969032    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/0.log"
Nov 01 03:42:05.969456 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:05.969076    2583 generic.go:356] "Generic (PLEG): container finished" podID="b5c8ccd8-df33-4df9-8311-5a163e8e31b3" containerID="c68ac2018baf8dfe5f4e94c194be439e50c3fd2313163064483b098de1262bb1" exitCode=255
Nov 01 03:42:05.969456 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:05.969171    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n" event={"ID":"b5c8ccd8-df33-4df9-8311-5a163e8e31b3","Type":"ContainerDied","Data":"c68ac2018baf8dfe5f4e94c194be439e50c3fd2313163064483b098de1262bb1"}
Nov 01 03:42:05.969456 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:05.969367    2583 scope.go:122] "RemoveContainer" containerID="c68ac2018baf8dfe5f4e94c194be439e50c3fd2313163064483b098de1262bb1"
Nov 01 03:42:06.973562 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:06.973532    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-74gj4" event={"ID":"f4070d3e-397e-4202-abb8-ff360ea75ee2","Type":"ContainerStarted","Data":"c896df2f02a0172275eb17f57f4a00aab9422a8fd034343d8be28d92a13fae1c"}
Nov 01 03:42:06.975152 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:06.975131    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:42:06.975548 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:06.975532    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/0.log"
Nov 01 03:42:06.975602 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:06.975566    2583 generic.go:356] "Generic (PLEG): container finished" podID="b5c8ccd8-df33-4df9-8311-5a163e8e31b3" containerID="2f48c70a28d8e8414666c7e5b90f659b27ace8f85b98bbfcdb4f0f71bdf8242a" exitCode=255
Nov 01 03:42:06.975635 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:06.975595    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n" event={"ID":"b5c8ccd8-df33-4df9-8311-5a163e8e31b3","Type":"ContainerDied","Data":"2f48c70a28d8e8414666c7e5b90f659b27ace8f85b98bbfcdb4f0f71bdf8242a"}
Nov 01 03:42:06.975667 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:06.975649    2583 scope.go:122] "RemoveContainer" containerID="c68ac2018baf8dfe5f4e94c194be439e50c3fd2313163064483b098de1262bb1"
Nov 01 03:42:06.975841 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:06.975820    2583 scope.go:122] "RemoveContainer" containerID="2f48c70a28d8e8414666c7e5b90f659b27ace8f85b98bbfcdb4f0f71bdf8242a"
Nov 01 03:42:06.976025 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:06.976006    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-j945n_openshift-console-operator(b5c8ccd8-df33-4df9-8311-5a163e8e31b3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n" podUID="b5c8ccd8-df33-4df9-8311-5a163e8e31b3"
Nov 01 03:42:06.977228 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:06.977201    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-46gkp" event={"ID":"bd4e00b4-e891-4be2-b941-1238d43e8805","Type":"ContainerStarted","Data":"06ccb20d6615d9f9ff6de6cec025ef5f3b0cb8118ba3bd386ca0d8548082845d"}
Nov 01 03:42:07.081224 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.081193    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6j6bv"]
Nov 01 03:42:07.094313 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.094268    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-74gj4" podStartSLOduration=407.311017001 podStartE2EDuration="6m53.094254931s" totalImagesPullingTime="5.78323793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:07.094041639 +0000 UTC m=+39.797273489" watchObservedRunningTime="2026-11-01 03:42:07.094254931 +0000 UTC m=+39.797486779"
Nov 01 03:42:07.114555 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.114525    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:07.121118 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.121091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:07.121253 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.121185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:07.121310 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.121264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:07.121374 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.121337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:07.121459 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.121441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:07.121521 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.121488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:07.121574 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.121519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:07.121574 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.121565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:42:07.121706 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.121691    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:42:07.121770 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.121758    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert podName:5cf6ba1f-8ecb-46fd-a2b9-db4319378237 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:15.1217353 +0000 UTC m=+47.824967143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2bnwm" (UID: "5cf6ba1f-8ecb-46fd-a2b9-db4319378237") : secret "networking-console-plugin-cert" not found
Nov 01 03:42:07.122224 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122202    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:07.122224 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122221    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565fcbb6bc-wvnqs: secret "image-registry-tls" not found
Nov 01 03:42:07.122358 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122261    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls podName:c7c51022-86cd-4cac-bee9-22a33fd53e22 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:15.122248834 +0000 UTC m=+47.825480675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls") pod "image-registry-565fcbb6bc-wvnqs" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22") : secret "image-registry-tls" not found
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122589    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122604    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8894dd498-k59gh: secret "image-registry-tls" not found
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122631    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122642    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls podName:6979e621-5326-4fff-8641-af89a3065735 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:15.122627807 +0000 UTC m=+47.825859648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls") pod "image-registry-8894dd498-k59gh" (UID: "6979e621-5326-4fff-8641-af89a3065735") : secret "image-registry-tls" not found
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122680    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls podName:19a62e18-e602-4b06-ae13-49276b60f80c nodeName:}" failed. No retries permitted until 2026-11-01 03:42:15.122665154 +0000 UTC m=+47.825897001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls") pod "dns-default-7v4p5" (UID: "19a62e18-e602-4b06-ae13-49276b60f80c") : secret "dns-default-metrics-tls" not found
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122634    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122705    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122718    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:15.122709583 +0000 UTC m=+47.825941411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : secret "router-metrics-certs-default" not found
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122739    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls podName:91524329-4a22-4667-bdeb-fd72ef337912 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:15.122730227 +0000 UTC m=+47.825962078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zvcfn" (UID: "91524329-4a22-4667-bdeb-fd72ef337912") : secret "samples-operator-tls" not found
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122798    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:15.12278856 +0000 UTC m=+47.826020420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : configmap references non-existent config key: service-ca.crt
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122850    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:07.122904 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.122882    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls podName:d50a0b39-914c-4b62-9b0b-4c6ab787f9d7 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:15.122872601 +0000 UTC m=+47.826104455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pknwh" (UID: "d50a0b39-914c-4b62-9b0b-4c6ab787f9d7") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:07.145973 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.145948    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6j6bv"]
Nov 01 03:42:07.170563 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.170539    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6nkgv\""
Nov 01 03:42:07.170703 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.170538    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 03:42:07.170703 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.170595    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 03:42:07.170703 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.170694    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 03:42:07.170831 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.170704    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 03:42:07.222891 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.222864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-key\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:07.223090 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.223069    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-cabundle\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:07.223161 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.223105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ds28\" (UniqueName: \"kubernetes.io/projected/0207b2e9-f007-41a7-96c3-aeecde61c873-kube-api-access-9ds28\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:07.324238 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.324160    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-key\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:07.324387 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.324319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-cabundle\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:07.324387 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.324346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ds28\" (UniqueName: \"kubernetes.io/projected/0207b2e9-f007-41a7-96c3-aeecde61c873-kube-api-access-9ds28\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:07.325148 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.325126    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-cabundle\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:07.326962 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.326942    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-key\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:07.337899 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.337855    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-46gkp" podStartSLOduration=2.784544777 podStartE2EDuration="8.337843469s" totalImagesPullingTime="5.553298692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:07.337673224 +0000 UTC m=+40.040905074" watchObservedRunningTime="2026-11-01 03:42:07.337843469 +0000 UTC m=+40.041075318"
Nov 01 03:42:07.361363 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.361333    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7rft8"]
Nov 01 03:42:07.381517 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.381492    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.388358 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.388334    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ds28\" (UniqueName: \"kubernetes.io/projected/0207b2e9-f007-41a7-96c3-aeecde61c873-kube-api-access-9ds28\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:07.420344 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.420318    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7rft8"]
Nov 01 03:42:07.436095 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.436070    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 03:42:07.436220 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.436110    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 03:42:07.436220 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.436215    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8fn5p\""
Nov 01 03:42:07.450875 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.450066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:07.525815 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.525716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d1984d7-7b8d-40a3-b464-ba27be829957-crio-socket\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.525815 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.525786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d1984d7-7b8d-40a3-b464-ba27be829957-data-volume\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.526057 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.526026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.526257 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.526200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jf8c\" (UniqueName: \"kubernetes.io/projected/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-api-access-8jf8c\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.526409 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.526388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.627822 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.627122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d1984d7-7b8d-40a3-b464-ba27be829957-crio-socket\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.627822 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.627198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d1984d7-7b8d-40a3-b464-ba27be829957-data-volume\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.627822 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.627235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.627822 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.627251    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d1984d7-7b8d-40a3-b464-ba27be829957-crio-socket\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.627822 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.627265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jf8c\" (UniqueName: \"kubernetes.io/projected/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-api-access-8jf8c\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.627822 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.627375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.627822 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.627511    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:07.627822 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.627569    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:08.127551335 +0000 UTC m=+40.830783178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:07.627822 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.627755    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d1984d7-7b8d-40a3-b464-ba27be829957-data-volume\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.627822 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.627768    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:42:07.627822 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.627810    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:08.127794869 +0000 UTC m=+40.831026716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : configmap "kube-rbac-proxy" not found
Nov 01 03:42:07.691001 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.690967    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jf8c\" (UniqueName: \"kubernetes.io/projected/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-api-access-8jf8c\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:07.853545 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.853515    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6j6bv"]
Nov 01 03:42:07.857100 ip-10-0-129-252 kubenswrapper[2583]: W1101 03:42:07.857069    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0207b2e9_f007_41a7_96c3_aeecde61c873.slice/crio-8a19789d7e1d51d9a6473fae84b7236859b11f19613d2cdadd271858c1a87fd1 WatchSource:0}: Error finding container 8a19789d7e1d51d9a6473fae84b7236859b11f19613d2cdadd271858c1a87fd1: Status 404 returned error can't find the container with id 8a19789d7e1d51d9a6473fae84b7236859b11f19613d2cdadd271858c1a87fd1
Nov 01 03:42:07.983780 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.983753    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:42:07.984241 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.984223    2583 scope.go:122] "RemoveContainer" containerID="2f48c70a28d8e8414666c7e5b90f659b27ace8f85b98bbfcdb4f0f71bdf8242a"
Nov 01 03:42:07.984472 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:07.984451    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-j945n_openshift-console-operator(b5c8ccd8-df33-4df9-8311-5a163e8e31b3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n" podUID="b5c8ccd8-df33-4df9-8311-5a163e8e31b3"
Nov 01 03:42:07.986814 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.986787    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv" event={"ID":"0207b2e9-f007-41a7-96c3-aeecde61c873","Type":"ContainerStarted","Data":"bb06ad31e5490c03d86f1c65996f5955e59649415125c3106230a92f188e8654"}
Nov 01 03:42:07.986933 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:07.986840    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv" event={"ID":"0207b2e9-f007-41a7-96c3-aeecde61c873","Type":"ContainerStarted","Data":"8a19789d7e1d51d9a6473fae84b7236859b11f19613d2cdadd271858c1a87fd1"}
Nov 01 03:42:08.131734 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:08.131504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:08.131734 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:08.131659    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:08.131734 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:08.131733    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:09.131709935 +0000 UTC m=+41.834941840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:08.132126 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:08.132097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:08.132325 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:08.132307    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:42:08.132398 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:08.132360    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:09.132344764 +0000 UTC m=+41.835576597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : configmap "kube-rbac-proxy" not found
Nov 01 03:42:08.227010 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:08.226953    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv" podStartSLOduration=1.226935828 podStartE2EDuration="1.226935828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:08.225885872 +0000 UTC m=+40.929117724" watchObservedRunningTime="2026-11-01 03:42:08.226935828 +0000 UTC m=+40.930167679"
Nov 01 03:42:09.142376 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:09.142334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:09.142862 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:09.142466    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:42:09.142862 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:09.142529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:09.142862 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:09.142545    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:11.142523444 +0000 UTC m=+43.845755286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : configmap "kube-rbac-proxy" not found
Nov 01 03:42:09.142862 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:09.142610    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:09.142862 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:09.142671    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:11.142656033 +0000 UTC m=+43.845887874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:09.643061 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:09.643016    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:09.643272 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:09.643095    2583 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:09.643544 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:09.643526    2583 scope.go:122] "RemoveContainer" containerID="2f48c70a28d8e8414666c7e5b90f659b27ace8f85b98bbfcdb4f0f71bdf8242a"
Nov 01 03:42:09.643777 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:09.643752    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-j945n_openshift-console-operator(b5c8ccd8-df33-4df9-8311-5a163e8e31b3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n" podUID="b5c8ccd8-df33-4df9-8311-5a163e8e31b3"
Nov 01 03:42:09.995134 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:09.995095    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hbgqw" event={"ID":"42ba278c-e933-4575-872b-3420353c400a","Type":"ContainerStarted","Data":"7d764601a97a0fcd1fb7e9e924e3b5aee869c00d3c93343346c4462cc3d428c0"}
Nov 01 03:42:09.995938 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:09.995920    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:42:11.163313 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:11.163289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:11.163634 ip-10-0-129-252 kubenswrapper[2583]: I1101 03:42:11.163413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:11.163634 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:11.163485    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:11.163634 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:11.163545    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:42:11.163634 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:11.163578    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:15.163562571 +0000 UTC m=+47.866794401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:11.163634 ip-10-0-129-252 kubenswrapper[2583]: E1101 03:42:11.163598    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:15.163587928 +0000 UTC m=+47.866819754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : configmap "kube-rbac-proxy" not found
Nov 01 03:42:11.347459 ip-10-0-129-252 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 03:42:11.359887 ip-10-0-129-252 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 03:42:11.360110 ip-10-0-129-252 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 03:42:11.360167 ip-10-0-129-252 systemd[1]: kubelet.service: Consumed 2.938s CPU time.
Nov 01 03:42:11.372146 ip-10-0-129-252 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 03:42:11.490587 ip-10-0-129-252 kubenswrapper[4882]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:42:11.490587 ip-10-0-129-252 kubenswrapper[4882]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 03:42:11.490587 ip-10-0-129-252 kubenswrapper[4882]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:42:11.490587 ip-10-0-129-252 kubenswrapper[4882]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:42:11.494046 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.493953    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:42:11.494046 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494042    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:42:11.494046 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494046    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:42:11.494046 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494050    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:42:11.494046 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494053    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:42:11.494046 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494057    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:42:11.494046 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494060    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494063    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494067    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494071    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494075    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494080    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494084    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494087    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494090    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494092    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494095    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494097    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494100    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494103    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494106    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494109    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494111    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494114    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494116    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:42:11.494346 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494121    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494125    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494128    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494130    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494133    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494136    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494139    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494141    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494144    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494147    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494151    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494156    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494161    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494165    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494168    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494171    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494174    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494177    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494179    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:42:11.494913 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494182    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494185    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494187    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494190    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494195    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494199    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494205    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494209    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494212    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494215    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494218    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494221    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494224    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494227    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494233    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494237    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494241    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494246    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494251    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:42:11.495538 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494254    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494258    4882 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494262    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494266    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494270    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494273    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494276    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494279    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494281    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494285    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494288    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494290    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494293    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494295    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494298    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494301    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494304    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494307    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494310    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:42:11.496107 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494312    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494315    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494318    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494322    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494326    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494330    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494334    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494338    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494343    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494348    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494351    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494355    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494358    4882 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494361    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494364    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494366    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494369    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494372    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494375    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494377    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:42:11.496719 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494380    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494382    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494389    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.494393    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495287    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495300    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495306    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495311    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495315    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495320    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495325    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495329    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495333    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495338    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495343    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495348    4882 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495352    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495355    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495357    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:42:11.497304 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495362    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495366    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495370    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495375    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495379    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495385    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495389    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495394    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495399    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495403    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495408    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495412    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495416    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495420    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495439    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495444    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495448    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495453    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495457    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:42:11.497885 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495462    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495467    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495471    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495476    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495482    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495486    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495491    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495496    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495500    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495505    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495511    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495514    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495517    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495520    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495522    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495525    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495527    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495530    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495533    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:42:11.498454 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495535    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495538    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495541    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495544    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495547    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495549    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495552    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495555    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495558    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495562    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495564    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495567    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495570    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495573    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495575    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495578    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495580    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495582    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495585    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495588    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:42:11.498986 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495590    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495593    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495595    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495598    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495601    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495604    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495606    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495610    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495613    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495615    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495618    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495620    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495622    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495625    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495628    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495631    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495633    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495636    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495638    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495641    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:42:11.499620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495643    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495646    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495650    4882 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495654    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495658    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495662    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495667    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495671    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495675    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495680    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495685    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495689    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.495696    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496322    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496334    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496342    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496347    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496352    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496356    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:42:11.500255 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496361    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496366    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496371    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496375    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496380    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496385    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496389    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496394    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496398    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496402    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496407    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496411    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496416    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496421    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496438    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496444    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496448    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496452    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496455    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496458    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:42:11.500830 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496461    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496463    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496467    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496469    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496472    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496475    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496477    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496481    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496483    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496488    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496492    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496498    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496504    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496509    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496515    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496519    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496523    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496528    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:42:11.501495 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496533    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496538    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496543    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496547    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496552    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496555    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496558    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496562    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496566    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496571    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496576    4882 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496578    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496581    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496584    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496586    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496589    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496592    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496594    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496597    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496600    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:42:11.502061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496602    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496605    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496608    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496610    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496615    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496618    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496620    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496623    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496625    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496628    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496632    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496637    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496641    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496645    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496649    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496654    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496658    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496663    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496667    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:42:11.502775 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496671    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496676    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496680    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496685    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496689    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496697    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496701    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496704    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496706    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496709    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496711    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496714    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496716    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496719    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496722    4882 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496725    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496728    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496731    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496735    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496738    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:42:11.503399 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496741    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496743    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.496746    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496854    4882 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496865    4882 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496874    4882 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496878    4882 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496883    4882 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496887    4882 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496892    4882 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496896    4882 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496899    4882 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496903    4882 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496906    4882 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496909    4882 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496912    4882 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496915    4882 flags.go:64] FLAG: --cgroup-root=""
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496921    4882 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496924    4882 flags.go:64] FLAG: --client-ca-file=""
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496928    4882 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496931    4882 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496935    4882 flags.go:64] FLAG: --cluster-domain=""
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496938    4882 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496941    4882 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 03:42:11.503998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496944    4882 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496947    4882 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496956    4882 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496959    4882 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496962    4882 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496967    4882 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496972    4882 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496977    4882 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496982    4882 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496988    4882 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.496993    4882 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497003    4882 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497008    4882 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497013    4882 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497018    4882 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497022    4882 flags.go:64] FLAG: --enable-server="true"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497026    4882 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497031    4882 flags.go:64] FLAG: --event-burst="100"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497034    4882 flags.go:64] FLAG: --event-qps="50"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497037    4882 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497040    4882 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497043    4882 flags.go:64] FLAG: --eviction-hard=""
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497047    4882 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497050    4882 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497053    4882 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 03:42:11.504841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497058    4882 flags.go:64] FLAG: --eviction-soft=""
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497061    4882 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497064    4882 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497067    4882 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497070    4882 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497073    4882 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497076    4882 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497079    4882 flags.go:64] FLAG: --feature-gates=""
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497083    4882 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497086    4882 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497089    4882 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497092    4882 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497095    4882 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497098    4882 flags.go:64] FLAG: --help="false"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497101    4882 flags.go:64] FLAG: --hostname-override="ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497105    4882 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497108    4882 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497111    4882 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497120    4882 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497126    4882 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497131    4882 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497137    4882 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497141    4882 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 03:42:11.505641 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497146    4882 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497149    4882 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497152    4882 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497156    4882 flags.go:64] FLAG: --kube-reserved=""
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497159    4882 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497162    4882 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497165    4882 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497170    4882 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497174    4882 flags.go:64] FLAG: --lock-file=""
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497181    4882 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497187    4882 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497192    4882 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497200    4882 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497205    4882 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497210    4882 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497215    4882 flags.go:64] FLAG: --logging-format="text"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497219    4882 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497222    4882 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497225    4882 flags.go:64] FLAG: --manifest-url=""
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497228    4882 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497238    4882 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497241    4882 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497245    4882 flags.go:64] FLAG: --max-pods="110"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497248    4882 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497251    4882 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 03:42:11.506336 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497254    4882 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497257    4882 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497262    4882 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497265    4882 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497268    4882 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497275    4882 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497280    4882 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497284    4882 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497289    4882 flags.go:64] FLAG: --pod-cidr=""
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497294    4882 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497299    4882 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497304    4882 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497309    4882 flags.go:64] FLAG: --port="10250"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497314    4882 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497319    4882 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-01677bd4104f0690c"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497323    4882 flags.go:64] FLAG: --qos-reserved=""
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497328    4882 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497331    4882 flags.go:64] FLAG: --register-node="true"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497336    4882 flags.go:64] FLAG: --register-with-taints=""
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497341    4882 flags.go:64] FLAG: --registry-burst="10"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497346    4882 flags.go:64] FLAG: --registry-qps="5"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497351    4882 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497355    4882 flags.go:64] FLAG: --reserved-memory=""
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497361    4882 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497367    4882 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497371    4882 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 03:42:11.507187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497376    4882 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497381    4882 flags.go:64] FLAG: --runonce="false"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497385    4882 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497388    4882 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497391    4882 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497394    4882 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497397    4882 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497401    4882 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497406    4882 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497409    4882 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497412    4882 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497415    4882 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497417    4882 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497420    4882 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497423    4882 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497438    4882 flags.go:64] FLAG: --system-cgroups=""
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497442    4882 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497449    4882 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497454    4882 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497459    4882 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497466    4882 flags.go:64] FLAG: --tls-min-version=""
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497471    4882 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497478    4882 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497482    4882 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497485    4882 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 03:42:11.508232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497488    4882 flags.go:64] FLAG: --v="2"
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497493    4882 flags.go:64] FLAG: --version="false"
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497497    4882 flags.go:64] FLAG: --vmodule=""
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497501    4882 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.497505    4882 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497683    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497690    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497695    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497699    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497704    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497709    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497712    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497715    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497718    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497721    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497727    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497730    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497733    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497735    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497738    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497741    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:42:11.509148 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497743    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497746    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497749    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497751    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497754    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497763    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497766    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497769    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497775    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497780    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497784    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497789    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497791    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497796    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497799    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497803    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497806    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497812    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497819    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:42:11.510006 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497824    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497828    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497832    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497835    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497838    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497840    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497843    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497847    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497856    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497860    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497864    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497869    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497872    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497877    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497881    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497884    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497887    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497889    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497893    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:42:11.510728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497895    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497898    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497903    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497906    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497908    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497911    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497913    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497916    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497918    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497921    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497924    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497927    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497929    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497932    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497935    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497937    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497940    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497942    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497945    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497947    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:42:11.511365 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497951    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497953    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497961    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497964    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497967    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497970    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497972    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497975    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497979    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497983    4882 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497987    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497993    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.497997    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498002    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498007    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498010    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498013    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498016    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498018    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498021    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:42:11.512036 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498023    4882 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498026    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498030    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498034    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498038    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498043    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498048    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498052    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498056    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498060    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498064    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498068    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.498098    4882 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498135    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498140    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498145    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:42:11.512685 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498148    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498151    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498154    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498158    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498162    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498167    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498172    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498177    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498181    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498184    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498187    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498192    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498194    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498198    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498202    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498206    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498210    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498214    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498218    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498223    4882 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:42:11.513153 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498227    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498231    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498233    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498236    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498238    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498241    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498243    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498246    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498249    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498252    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498254    4882 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498257    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498261    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498264    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498266    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498269    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498272    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498275    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498277    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498280    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:42:11.514211 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498282    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498285    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498288    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498292    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498294    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498297    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498300    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498302    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498307    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498311    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498317    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498322    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498327    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498331    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498336    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498339    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498341    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498344    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498347    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498349    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:42:11.514949 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498352    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498358    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498362    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498366    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498370    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498375    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498379    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498383    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498387    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498391    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498395    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498401    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498406    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498410    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498414    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498422    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498440    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498445    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:42:11.515537 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498450    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498456    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498459    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498463    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498466    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498470    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498475    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498480    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498484    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498488    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498492    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498495    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498497    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498502    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498505    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498509    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498513    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498518    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498524    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:42:11.516080 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498528    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498531    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498534    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498537    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498540    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.498543    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.502797    4882 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.502817    4882 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.502834    4882 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.502839    4882 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502910    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502916    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502920    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502923    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502926    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502928    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502931    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502934    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502937    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:42:11.516630 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502939    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502942    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502945    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502948    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502951    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502953    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502956    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502959    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502962    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502964    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502968    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502970    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502973    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502976    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502979    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502981    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502985    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502990    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502993    4882 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:42:11.517212 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502995    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.502999    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503002    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503004    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503007    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503010    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503014    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503019    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503024    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503028    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503031    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503034    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503037    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503039    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503043    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503047    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503050    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503053    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503056    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:42:11.517944 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503058    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503061    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503064    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503067    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503069    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503072    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503075    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503077    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503080    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503082    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503085    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503088    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503090    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503093    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503095    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503100    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503104    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503108    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503112    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:42:11.518570 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503116    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503120    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503124    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503128    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503132    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503136    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503141    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503145    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503150    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503154    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503158    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503163    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503167    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503172    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503176    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503179    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503183    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503185    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503188    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503191    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:42:11.519163 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503193    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503196    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503198    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503201    4882 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503204    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503206    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503209    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503211    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503214    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503217    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503220    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503222    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503225    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503228    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503230    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503233    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503236    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503238    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503241    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:42:11.519840 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503244    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.503282    4882 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503496    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503503    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503506    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503509    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503511    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503514    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503517    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503519    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503522    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503525    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503528    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503533    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503537    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503541    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:42:11.520374 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503545    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503549    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503553    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503557    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503561    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503565    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503569    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503573    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503576    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503581    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503585    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503590    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503595    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503600    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503605    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503609    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503612    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503614    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503617    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:42:11.520870 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503620    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503622    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503625    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503628    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503630    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503633    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503636    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503640    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503644    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503647    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503649    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503652    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503655    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503657    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503660    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503664    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503668    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503672    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503676    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:42:11.521550 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503680    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503684    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503688    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503692    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503696    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503700    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503705    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503708    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503713    4882 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503717    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503722    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503727    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503731    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503736    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503742    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503746    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503749    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503752    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503755    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503757    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:42:11.522224 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503760    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503763    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503765    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503769    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503772    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503774    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503777    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503779    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503782    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503784    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503787    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503790    4882 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503792    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503794    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503798    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503802    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503806    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503809    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503812    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503815    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:42:11.523070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503819    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503823    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503827    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503831    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503835    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503842    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503846    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503849    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503854    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503858    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503860    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503863    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503866    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503868    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.503896    4882 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:42:11.523994 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503935    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503940    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503944    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503947    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503950    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503953    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503955    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503959    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503964    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503968    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503972    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503976    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503979    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503986    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503990    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503994    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.503998    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504001    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504004    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:42:11.524664 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504007    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504010    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504012    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504016    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504019    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504022    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504025    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504028    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504030    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504033    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504036    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504039    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504042    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504044    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504047    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504049    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504052    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504054    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504057    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504060    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:42:11.525477 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504062    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504065    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504067    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504070    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504073    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504075    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504081    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504084    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504087    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504089    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504092    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504095    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504100    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504104    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504108    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504114    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504119    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504122    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504124    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:42:11.526339 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504127    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504130    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504132    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504136    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504140    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504144    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504147    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504150    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504153    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504156    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504159    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504161    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504164    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504166    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504169    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504172    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504174    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504177    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504179    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:42:11.527074 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504183    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504186    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504190    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504194    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504199    4882 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504203    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504208    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504212    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504218    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504225    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504229    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504234    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504238    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504241    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504244    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504247    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504249    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504252    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504255    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504258    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:42:11.527764 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504261    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504263    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504266    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504268    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504271    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504274    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504277    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504279    4882 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:11.504282    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.504352    4882 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.505458    4882 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.505522    4882 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.505771    4882 server.go:1014] "Starting client certificate rotation"
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.505878    4882 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.505921    4882 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 03:36:28 +0000 UTC" deadline="2028-06-08 22:20:52.75631839 +0000 UTC"
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.505952    4882 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14058h38m41.250369546s"
Nov 01 03:42:11.528608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.506370    4882 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:42:11.529188 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.507602    4882 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:42:11.529188 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.510635    4882 log.go:25] "Validated CRI v1 runtime API"
Nov 01 03:42:11.529188 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.513138    4882 log.go:25] "Validated CRI v1 image API"
Nov 01 03:42:11.529188 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.514072    4882 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 03:42:11.529188 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.516756    4882 fs.go:135] Filesystem UUIDs: map[4539a4fa-ff2f-412d-aa88-0239caed6907:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7b0db276-8dcb-45d7-8dbc-510c220568f7:/dev/nvme0n1p3]
Nov 01 03:42:11.529394 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.516776    4882 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e2eeb1496827ebfe8a3d759fbde6b886d3bfc63aa5a41eb4c005da35e24511c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e2eeb1496827ebfe8a3d759fbde6b886d3bfc63aa5a41eb4c005da35e24511c/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2508a5e206e9538f252ec6fe28ebb3026b5f5208ff47d6befd52c57ac618fe80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2508a5e206e9538f252ec6fe28ebb3026b5f5208ff47d6befd52c57ac618fe80/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45e31a093acba39081e9a7a984525dbc3f7e09bf532dfb053c963b0872613f55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45e31a093acba39081e9a7a984525dbc3f7e09bf532dfb053c963b0872613f55/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5333dc8a413658f507ade37494d3b35e2d7b13b003e407dcceb2d358e579cbcb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5333dc8a413658f507ade37494d3b35e2d7b13b003e407dcceb2d358e579cbcb/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57ff89a064810529b94be1b6ba9e0dd51706a0c0d1f572b52d1b3988334a5df4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57ff89a064810529b94be1b6ba9e0dd51706a0c0d1f572b52d1b3988334a5df4/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/599b58367432f5e13abaea4a8c58ff70eafe140d71e4db85b9d8b343f38ff892/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/599b58367432f5e13abaea4a8c58ff70eafe140d71e4db85b9d8b343f38ff892/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61008947d7850e6869cb38b5a0c920216d88e89cdf30c870a696cdce60662848/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61008947d7850e6869cb38b5a0c920216d88e89cdf30c870a696cdce60662848/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7969e4f0dc467b9847fbb4383dda48e41a9d3b0bbbb3128df6cdbb1beed0063f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7969e4f0dc467b9847fbb4383dda48e41a9d3b0bbbb3128df6cdbb1beed0063f/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/821b9ce2527df5ee5e49cb7f6f27a60a82b469a8aa7c2c188e4ee93c8f28673d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/821b9ce2527df5ee5e49cb7f6f27a60a82b469a8aa7c2c188e4ee93c8f28673d/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a19789d7e1d51d9a6473fae84b7236859b11f19613d2cdadd271858c1a87fd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a19789d7e1d51d9a6473fae84b7236859b11f19613d2cdadd271858c1a87fd1/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8eb74a593784fbcf0b3f74c2d761707f1a43a018745a6a759191727a4b627f98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8eb74a593784fbcf0b3f74c2d761707f1a43a018745a6a759191727a4b627f98/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a302cc5ca4e5b357243b484318ba248f37ee25402ccb51fe546813105e745e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a302cc5ca4e5b357243b484318ba248f37ee25402ccb51fe546813105e745e7/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3f7f09bb225b48f6c9a5c9f2adc869de435152a098c04e5c0a484be56f891e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3f7f09bb225b48f6c9a5c9f2adc869de435152a098c04e5c0a484be56f891e5/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acdfc83ea304eb9b14652fcaedebc62bcc8c79257d72795c3dccd900e684c119/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acdfc83ea304eb9b14652fcaedebc62bcc8c79257d72795c3dccd900e684c119/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0f5216e5bbf8b41fc984fc29d3fca32dd91c056efedd73158eb328680f5344a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0f5216e5bbf8b41fc984fc29d3fca32dd91c056efedd73158eb328680f5344a/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6f1b2cd380df1338c03abfd519930a57f929e67f5dc36b6ee3a5fdaa693207f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6f1b2cd380df1338c03abfd519930a57f929e67f5dc36b6ee3a5fdaa693207f/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b97a30eaa2bd4a1e67bfb448f070dd06f8e4387d1643f701058178d9fa617db3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b97a30eaa2bd4a1e67bfb448f070dd06f8e4387d1643f701058178d9fa617db3/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c43b6b72bd59316f0febc6194f4dcdd821474d757ab5392f96bd9a800c60abe6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c43b6b72bd59316f0febc6194f4dcdd821474d757ab5392f96bd9a800c60abe6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2739256e3f744805bb0bdff60b5cd89a58785399ab8cf2bc66bdedd3cf016ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2739256e3f744805bb0bdff60b5cd89a58785399ab8cf2bc66bdedd3cf016ae/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6a1497a78002a8af458579b2c7a0bfba8207fe50ab4a01890986e51cbee0832/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6a1497a78002a8af458579b2c7a0bfba8207fe50ab4a01890986e51cbee0832/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea73befeff421f14d4c19072786a918dbffe0de1b2fa777b588de37298efbbba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea73befeff421f14d4c19072786a918dbffe0de1b2fa777b588de37298efbbba/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f21616762aa00d7dc8274121d13f5eb93b48968cc729caf759e7348edda9b227/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f21616762aa00d7dc8274121d13f5eb93b48968cc729caf759e7348edda9b227/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f249a50d3760b1f2cd241f459207c71c50bf8e97cecc190068a2042afaf009a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f249a50d3760b1f2cd241f459207c71c50bf8e97cecc190068a2042afaf009a2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0207b2e9-f007-41a7-96c3-aeecde61c873/volumes/kubernetes.io~projected/kube-api-access-9ds28:{mountpoint:/var/lib/kubelet/pods/0207b2e9-f007-41a7-96c3-aeecde61c873/volumes/kubernetes.io~projected/kube-api-access-9ds28 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0207b2e9-f007-41a7-96c3-aeecde61c873/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/0207b2e9-f007-41a7-96c3-aeecde61c873/volumes/kubernetes.io~secret/signing-key major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0271bbb9-50cc-47e2-8681-20e6b2550469/volumes/kubernetes.io~projected/kube-api-access-rqmbq:{mountpoint:/var/lib/kubelet/pods/0271bbb9-50cc-47e2-8681-20e6b2550469/volumes/kubernetes.io~projected/kube-api-access-rqmbq major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0271bbb9-50cc-47e2-8681-20e6b2550469/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0271bbb9-50cc-47e2-8681-20e6b2550469/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/036c1722-9db5-406e-bc09-0aa6f215e1e0/volumes/kubernetes.io~projected/kube-api-access-4n9m2:{mountpoint:/var/lib/kubelet/pods/036c1722-9db5-406e-bc09-0aa6f215e1e0/volumes/kubernetes.io~projected/kube-api-access-4n9m2 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/036c1722-9db5-406e-bc09-0aa6f215e1e0/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/036c1722-9db5-406e-bc09-0aa6f215e1e0/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/036c1722-9db5-406e-bc09-0aa6f215e1e0/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/036c1722-9db5-406e-bc09-0aa6f215e1e0/volumes/kubernetes.io~secret/stats-auth major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19a62e18-e602-4b06-ae13-49276b60f80c/volumes/kubernetes.io~projected/kube-api-access-2c8lp:{mountpoint:/var/lib/kubelet/pods/19a62e18-e602-4b06-ae13-49276b60f80c/volumes/kubernetes.io~projected/kube-api-access-2c8lp major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1984d7-7b8d-40a3-b464-ba27be829957/volumes/kubernetes.io~projected/kube-api-access-8jf8c:{mountpoint:/var/lib/kubelet/pods/1d1984d7-7b8d-40a3-b464-ba27be829957/volumes/kubernetes.io~projected/kube-api-access-8jf8c major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20c62809-5ccf-419d-afae-8d140f67309d/volumes/kubernetes.io~projected/kube-api-access-g82ht:{mountpoint:/var/lib/kubelet/pods/20c62809-5ccf-419d-afae-8d140f67309d/volumes/kubernetes.io~projected/kube-api-access-g82ht major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cd84ba1-3255-444e-b2e0-2a3d4ac25988/volumes/kubernetes.io~projected/kube-api-access-zc4z5:{mountpoint:/var/lib/kubelet/pods/3cd84ba1-3255-444e-b2e0-2a3d4ac25988/volumes/kubernetes.io~projected/kube-api-access-zc4z5 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cd84ba1-3255-444e-b2e0-2a3d4ac25988/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3cd84ba1-3255-444e-b2e0-2a3d4ac25988/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42ba278c-e933-4575-872b-3420353c400a/volumes/kubernetes.io~projected/kube-api-access-4gcbx:{mountpoint:/var/lib/kubelet/pods/42ba278c-e933-4575-872b-3420353c400a/volumes/kubernetes.io~projected/kube-api-access-4gcbx major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b07433f-bfec-4e99-97e4-1af1646c45ac/volumes/kubernetes.io~projected/kube-api-access-wbzqc:{mountpoint:/var/lib/kubelet/pods/5b07433f-bfec-4e99-97e4-1af1646c45ac/volumes/kubernetes.io~projected/kube-api-access-wbzqc major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5/volumes/kubernetes.io~projected/kube-api-access-htgqq:{mountpoint:/var/lib/kubelet/pods/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5/volumes/kubernetes.io~projected/kube-api-access-htgqq major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~projected/kube-api-access-kqjhm:{mountpoint:/var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~projected/kube-api-access-kqjhm major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10/volumes/kubernetes.io~projected/kube-api-access-b7vxk:{mountpoint:/var/lib/kubelet/pods/6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10/volumes/kubernetes.io~projected/kube-api-access-b7vxk major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89e71943-93a0-48b5-aa15-e060cc7b30f3/volumes/kubernetes.io~projected/kube-api-access-5pxlp:{mountpoint:/var/lib/kubelet/pods/89e71943-93a0-48b5-aa15-e060cc7b30f3/volumes/kubernetes.io~projected/kube-api-access-5pxlp major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d5bf13e-c653-45d9-85f2-a878a8de5236/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8d5bf13e-c653-45d9-85f2-a878a8de5236/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d5bf13e-c653-45d9-85f2-a878a8de5236/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8d5bf13e-c653-45d9-85f2-a878a8de5236/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d5bf13e-c653-45d9-85f2-a878a8de5236/volumes/kubernetes.io~projected/kube-api-access-bvfj2:{mountpoint:/var/lib/kubelet/pods/8d5bf13e-c653-45d9-85f2-a878a8de5236/volumes/kubernetes.io~projected/kube-api-access-bvfj2 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91524329-4a22-4667-bdeb-fd72ef337912/volumes/kubernetes.io~projected/kube-api-access-mctgm:{mountpoint:/var/lib/kubelet/pods/91524329-4a22-4667-bdeb-fd72ef337912/volumes/kubernetes.io~projected/kube-api-access-mctgm major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c80204c-9fa6-4c25-a1e6-737c7f217976/volumes/kubernetes.io~projected/kube-api-access-b92hs:{mountpoint:/var/lib/kubelet/pods/9c80204c-9fa6-4c25-a1e6-737c7f217976/volumes/kubernetes.io~projected/kube-api-access-b92hs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae127ade-3e32-4aa3-a528-324aa704bcdf/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ae127ade-3e32-4aa3-a528-324aa704bcdf/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3f80953-074f-455a-9c88-6b3154926050/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b3f80953-074f-455a-9c88-6b3154926050/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b496de82-bd77-42c5-8a29-e38514f99c5d/volumes/kubernetes.io~projected/kube-api-access-flxck:{mountpoint:/var/lib/kubelet/pods/b496de82-bd77-42c5-8a29-e38514f99c5d/volumes/kubernetes.io~projected/kube-api-access-flxck major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5c8ccd8-df33-4df9-8311-5a163e8e31b3/volumes/kubernetes.io~projected/kube-api-access-lqvdm:{mountpoint:/var/lib/kubelet/pods/b5c8ccd8-df33-4df9-8311-5a163e8e31b3/volumes/kubernetes.io~projected/kube-api-access-lqvdm major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5c8ccd8-df33-4df9-8311-5a163e8e31b3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b5c8ccd8-df33-4df9-8311-5a163e8e31b3/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba9103b6-e703-48e0-9276-aec7726fcd36/volumes/kubernetes.io~projected/kube-api-access-gpgbw:{mountpoint:/var/lib/kubelet/pods/ba9103b6-e703-48e0-9276-aec7726fcd36/volumes/kubernetes.io~projected/kube-api-access-gpgbw major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd4e00b4-e891-4be2-b941-1238d43e8805/volumes/kubernetes.io~projected/kube-api-access-vjqww:{mountpoint:/var/lib/kubelet/pods/bd4e00b4-e891-4be2-b941-1238d43e8805/volumes/kubernetes.io~projected/kube-api-access-vjqww major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd4e00b4-e891-4be2-b941-1238d43e8805/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/bd4e00b4-e891-4be2-b941-1238d43e8805/volumes/kubernetes.io~secret/cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~projected/kube-api-access-msg64:{mountpoint:/var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~projected/kube-api-access-msg64 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd1787d7-cd00-470d-b48f-c2d71e566dd2/volumes/kubernetes.io~projected/kube-api-access-b42gr:{mountpoint:/var/lib/kubelet/pods/cd1787d7-cd00-470d-b48f-c2d71e566dd2/volumes/kubernetes.io~projected/kube-api-access-b42gr major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd24318e-36ed-4fbb-9742-e6e2a9486283/volumes/kubernetes.io~projected/kube-api-access-z644b:{mountpoint:/var/lib/kubelet/pods/cd24318e-36ed-4fbb-9742-e6e2a9486283/volumes/kubernetes.io~projected/kube-api-access-z644b major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7/volumes/kubernetes.io~projected/kube-api-access-mfd98:{mountpoint:/var/lib/kubelet/pods/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7/volumes/kubernetes.io~projected/kube-api-access-mfd98 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f065d74a-fa1d-41f2-b4a9-6db03bd53a71/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f065d74a-fa1d-41f2-b4a9-6db03bd53a71/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f065d74a-fa1d-41f2-b4a9-6db03bd53a71/volumes/kubernetes.io~projected/kube-api-access-4fllm:{mountpoint:/var/lib/kubelet/pods/f065d74a-fa1d-41f2-b4a9-6db03bd53a71/volumes/kubernetes.io~projected/kube-api-access-4fllm major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f065d74a-fa1d-41f2-b4a9-6db03bd53a71/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f065d74a-fa1d-41f2-b4a9-6db03bd53a71/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4070d3e-397e-4202-abb8-ff360ea75ee2/volumes/kubernetes.io~projected/kube-api-access-gxjcw:{mountpoint:/var/lib/kubelet/pods/f4070d3e-397e-4202-abb8-ff360ea75ee2/volumes/kubernetes.io~projected/kube-api-access-gxjcw major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4070d3e-397e-4202-abb8-ff360ea75ee2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f4070d3e-397e-4202-abb8-ff360ea75ee2/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f74d5816-8e0a-4f49-83bd-5cd907a57cab/volumes/kubernetes.io~projected/kube-api-access-xsxnl:{mountpoint:/var/lib/kubelet/pods/f74d5816-8e0a-4f49-83bd-5cd907a57cab/volumes/kubernetes.io~projected/kube-api-access-xsxnl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9f6a2a3-88f9-4d37-b144-371147dd6a6e/volumes/kubernetes.io~projected/kube-api-access-hmxpq:{mountpoint:/var/lib/kubelet/pods/f9f6a2a3-88f9-4d37-b144-371147dd6a6e/volumes/kubernetes.io~projected/kube-api-access-hmxpq major:0 minor:222 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9393d4c14ef99d38c6f5d4750a07f1e7902d512cf092fa113c270f81594dc793/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/edf574c255877133833fb9e5158a35bbf47b8785c5f520d583362e6f4881d426/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/771375340c66f270b4e5abc2388670803a033c13e2f51a2ffe5f9344cf217f6f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/8d3241aea0b3e51ce2b33e794765d9291fdb2dcf76148f49380c5ebcf6042cfd/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/19edb17ace8966a9eccbdf427b8479383f3aab230be27520db035df86a2c00d9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/933d374a507bca2fe045bda9e3f9b2755fb3b3cfac0e64ea06a3e082a26b0c69/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3f34fefe75cdd0a544fe557d864ed891fd4fb2a8dba3a110e64e39f67cc4dded/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/560dacc926b8ec87fa25ee59b768ad170abc78adcb2ac52591847fa8ff166658/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/7340fe9840d42fb5f251751da1ca03b7dd3421aad27c2395a6093117cccdd661/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/42fafd25a5a8c3e49b9681722f4ce4a56f0fb61ef32545a8403a16fe510a173f/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/a5fe0720eb3e9caefae947ccd903888ed21e9b770440d04319abb77507cbf00d/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/6a0ab7ce7702e45a623475621b689cadcaa20b57dada85159e9ef52afa5dda87/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/05aa152fa67126ee12e011c228b0e6473c4c0e67f0d2d6c32d5062685c3fb785/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5a1377d45e7d883879e07b027a51e7571050a5fe677e34c1b9c8d33bd906a65b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f1afbf2eefceb7e31b6480dbaacf461e915235c0bd9e247022caabe3b240806f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/ba6f9b395b73479bb9d3497718a3c5ea65bb730202fe01001471cf35efeb3f27/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6128ce9ba49482bc3c486522e809fa84277ed91c9bbccd15ee4241c245d57063/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e086ae831c0cc83b02eb0b3354e6df774e5c1f43b93ec189077d93352406ae75/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/17a4b4aba609347bbaecc2160ab7d01851e7aeb2eb5d29b1c127e44c189834d2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/aa09042761e8ad910df99adb403198f16364632051ab80f871dc59a00ae80cd7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/de476a93a6dd1ed4d232a277e65821567874ea6703fcd858c2a9950f07284fa0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/552f1f8b1f261507a2c6e0c1c1d7b2cf6765bd4d5a62f37d345efd6265c9501e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/a13eea63d7f81cd87aea859c8d641888b453d6576f1db7901c5d4c576bb163c8/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/b1adf83d1b99405c267d989e16f1bac5f34ba6aa5f5d2cfa75b9f99a32d2d89e/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/eb2af1f21eded6408dbb18ec7b6fcc9082696f94a3cf83bbcf8f3d535697760b/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/aafa810e4cbbea3a2a1fe8b55c1310ed0a97ce0d01e2f780b81930e43073ac0e/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/cb9db7de67cc54f210c1eeb655bb11efee358b018bc8597061486f4badec1ebe/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/f91ef3b64a419590ede019661e7672b88d3a5a24653e31d166a61caebc13539b/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/ee7f684c2d54199c3d22cccbfb2b5ef093602564e8d28e23fc378b2295794edf/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/995e1370905e70cc4727afcfe93f1f0d298ad291fcf8c41f645e062111796dfc/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/cc3d4749dc49b858a821876e7e7cd41c340cce1b4775cf6e2ecf878c4d8d718a/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/c3dbccef7b8eef2962e09ad04810f4b981f47d423f93bad00476687d2ff59127/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/5701bcea2a286b1cb1f4ad4e380cc51c74740778c8d43fb50e9640895494c692/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/b47e30619df694779ca83f80efc24b58f627cef2adc5490744110924e5c2e22c/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/d1002a97c57b10493e6e86d84950e384283920100a677aa18dbdfbafac5e0dd5/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/2cdaccd6d1dde90f2ade9356732c705a03c925be44f5f6e7b4c7b9cd49a9e71d/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/f651c09e9a71596093f0f48ae938322ec318916b23f938eb4844da1b9eed535f/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/63e049536e1959efdbfa8824c8844043f1b6541fa340decdf19fae512490a3e8/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/993d0f5ac9c76e03740e0b0aa73ff66793161db23e589b334c0378911a682544/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/87f713eb7dee60b66ddd9e2930c729bc6436c55e76ea2af28bc10051be4dd7ef/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/b3769bf69eb04fde0d632447a9d59804e9ae31b68f686d6ea1638fbd40b884b1/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/76761ddb360cbbe08eda47aba3df8cf4eb9e4a99719ff886823d368b6dbabe6a/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/208b505f9146d8b492b4e66f8d4e8c70b4dae0e76f929571c55760e57f67bd30/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/2d29c9f0cdb9ffcd2abd109fff7f5c91dd9441d5959b50ffd4d42e61da956fd5/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/bd9338aad51212d7807113c079dd734f6db64a1a92eb87b1da32eb9335cded8d/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/aa1e27201b95f61fb042225cab1b929bd974e35f8881bba2bde05ee1ab7982e6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d26a6fa9db0ffe11c8c044e8a376cefdfacd6dd3481ba0f835cff5b215ae3a30/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/1a1f153245bf0b95c86ce42d5519ca2cbb3a7ad7bc07360a980dd0616fa943a0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/b1a15a1ea05ad46ef4aeadad554273bc7074d5901e0d58096d5902e2ed593ef3/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/79adcfb162e4c70dc7bf816456dd29a9f1e6e07f5bc05230a04d6f7fdd161a97/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0ea207d09753330396c0de9c3c665deb0248d7d7c99baea545a7f1eec0f40610/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e78ac3dd225a4ecef42411e1114380cda69626ca5485653500bdf7d15f1671ee/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/521f5c71af6e235090285f9efa5470633ae08fc727f7bfc9f5fc13c974d5f1e9/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/82599a956c557b710f291bf76637ed092bee8bb7583435096f1747663bb4b5d8/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.525310    4882 manager.go:217] Machine: {Timestamp:2026-11-01 03:42:11.524757878 +0000 UTC m=+0.086790887 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100731 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e7816fc5c295025e4fd9cfea46659 SystemUUID:ec2e7816-fc5c-2950-25e4-fd9cfea46659 BootID:d37aa4d8-0934-4a97-8c9c-2092319a0c49 Filesystems:[{Device:/var/lib/kubelet/pods/91524329-4a22-4667-bdeb-fd72ef337912/volumes/kubernetes.io~projected/kube-api-access-mctgm DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19a62e18-e602-4b06-ae13-49276b60f80c/volumes/kubernetes.io~projected/kube-api-access-2c8lp DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7/volumes/kubernetes.io~projected/kube-api-access-mfd98 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45e31a093acba39081e9a7a984525dbc3f7e09bf532dfb053c963b0872613f55/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d5bf13e-c653-45d9-85f2-a878a8de5236/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3cd84ba1-3255-444e-b2e0-2a3d4ac25988/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae127ade-3e32-4aa3-a528-324aa704bcdf/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba9103b6-e703-48e0-9276-aec7726fcd36/volumes/kubernetes.io~projected/kube-api-access-gpgbw DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c80204c-9fa6-4c25-a1e6-737c7f217976/volumes/kubernetes.io~projected/kube-api-access-b92hs DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5c8ccd8-df33-4df9-8311-5a163e8e31b3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/599b58367432f5e13abaea4a8c58ff70eafe140d71e4db85b9d8b343f38ff892/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4070d3e-397e-4202-abb8-ff360ea75ee2/volumes/kubernetes.io~projected/kube-api-access-gxjcw DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/036c1722-9db5-406e-bc09-0aa6f215e1e0/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~projected/kube-api-access-msg64 DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f065d74a-fa1d-41f2-b4a9-6db03bd53a71/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10/volumes/kubernetes.io~projected/kube-api-access-b7vxk DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61008947d7850e6869cb38b5a0c920216d88e89cdf30c870a696cdce60662848/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3f7f09bb225b48f6c9a5c9f2adc869de435152a098c04e5c0a484be56f891e5/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7969e4f0dc467b9847fbb4383dda48e41a9d3b0bbbb3128df6cdbb1beed0063f/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e2eeb1496827ebfe8a3d759fbde6b886d3bfc63aa5a41eb4c005da35e24511c/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d5bf13e-c653-45d9-85f2-a878a8de5236/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9f6a2a3-88f9-4d37-b144-371147dd6a6e/volumes/kubernetes.io~projected/kube-api-access-hmxpq DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/f065d74a-fa1d-41f2-b4a9-6db03bd53a71/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f065d74a-fa1d-41f2-b4a9-6db03bd53a71/volumes/kubernetes.io~projected/kube-api-access-4fllm DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0271bbb9-50cc-47e2-8681-20e6b2550469/volumes/kubernetes.io~projected/kube-api-access-rqmbq DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0207b2e9-f007-41a7-96c3-aeecde61c873/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/20c62809-5ccf-419d-afae-8d140f67309d/volumes/kubernetes.io~projected/kube-api-access-g82ht DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3cd84ba1-3255-444e-b2e0-2a3d4ac25988/volumes/kubernetes.io~projected/kube-api-access-zc4z5 DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5/volumes/kubernetes.io~projected/kube-api-access-htgqq DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57ff89a064810529b94be1b6ba9e0dd51706a0c0d1f572b52d1b3988334a5df4/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5333dc8a413658f507ade37494d3b35e2d7b13b003e407dcceb2d358e579cbcb/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b496de82-bd77-42c5-8a29-e38514f99c5d/volumes/kubernetes.io~projected/kube-api-access-flxck DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/036c1722-9db5-406e-bc09-0aa6f215e1e0/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd4e00b4-e891-4be2-b941-1238d43e8805/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f21616762aa00d7dc8274121d13f5eb93b48968cc729caf759e7348edda9b227/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/89e71943-93a0-48b5-aa15-e060cc7b30f3/volumes/kubernetes.io~projected/kube-api-access-5pxlp DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a302cc5ca4e5b357243b484318ba248f37ee25402ccb51fe546813105e745e7/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42ba278c-e933-4575-872b-3420353c400a/volumes/kubernetes.io~projected/kube-api-access-4gcbx DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/821b9ce2527df5ee5e49cb7f6f27a60a82b469a8aa7c2c188e4ee93c8f28673d/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2739256e3f744805bb0bdff60b5cd89a58785399ab8cf2bc66bdedd3cf016ae/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0f5216e5bbf8b41fc984fc29d3fca32dd91c056efedd73158eb328680f5344a/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b07433f-bfec-4e99-97e4-1af1646c45ac/volumes/kubernetes.io~projected/kube-api-access-wbzqc DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6f1b2cd380df1338c03abfd519930a57f929e67f5dc36b6ee3a5fdaa693207f/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6a1497a78002a8af458579b2c7a0bfba8207fe50ab4a01890986e51cbee0832/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d5bf13e-c653-45d9-85f2-a878a8de5236/volumes/kubernetes.io~projected/kube-api-access-bvfj2 DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2508a5e206e9538f252ec6fe28ebb3026b5f5208ff47d6befd52c57ac618fe80/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f249a50d3760b1f2cd241f459207c71c50bf8e97cecc190068a2042afaf009a2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd24318e-36ed-4fbb-9742-e6e2a9486283/volumes/kubernetes.io~projected/kube-api-access-z644b DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0207b2e9-f007-41a7-96c3-aeecde61c873/volumes/kubernetes.io~projected/kube-api-access-9ds28 DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acdfc83ea304eb9b14652fcaedebc62bcc8c79257d72795c3dccd900e684c119/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/036c1722-9db5-406e-bc09-0aa6f215e1e0/volumes/kubernetes.io~projected/kube-api-access-4n9m2 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a19789d7e1d51d9a6473fae84b7236859b11f19613d2cdadd271858c1a87fd1/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1984d7-7b8d-40a3-b464-ba27be829957/volumes/kubernetes.io~projected/kube-api-access-8jf8c DeviceMajor:0 DeviceMinor:324 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f74d5816-8e0a-4f49-83bd-5cd907a57cab/volumes/kubernetes.io~projected/kube-api-access-xsxnl DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3f80953-074f-455a-9c88-6b3154926050/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes/kubernetes.io~projected/kube-api-access-kqjhm DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8eb74a593784fbcf0b3f74c2d761707f1a43a018745a6a759191727a4b627f98/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c43b6b72bd59316f0febc6194f4dcdd821474d757ab5392f96bd9a800c60abe6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4070d3e-397e-4202-abb8-ff360ea75ee2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0271bbb9-50cc-47e2-8681-20e6b2550469/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd4e00b4-e891-4be2-b941-1238d43e8805/volumes/kubernetes.io~projected/kube-api-access-vjqww DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea73befeff421f14d4c19072786a918dbffe0de1b2fa777b588de37298efbbba/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5c8ccd8-df33-4df9-8311-5a163e8e31b3/volumes/kubernetes.io~projected/kube-api-access-lqvdm DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b97a30eaa2bd4a1e67bfb448f070dd06f8e4387d1643f701058178d9fa617db3/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd1787d7-cd00-470d-b48f-c2d71e566dd2/volumes/kubernetes.io~projected/kube-api-access-b42gr DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0e2eeb1496827eb MacAddress:36:70:a4:14:85:c5 Speed:10000 Mtu:8901} {Name:45e31a093acba39 MacAddress:3e:b0:8a:1c:50:7c Speed:10000 Mtu:8901} {Name:599b58367432f5e MacAddress:7e:fa:26:81:34:d2 Speed:10000 Mtu:8901} {Name:61008947d7850e6 MacAddress:96:90:ab:d7:88:c9 Speed:10000 Mtu:8901} {Name:7969e4f0dc467b9 MacAddress:ee:0d:eb:a8:30:9c Speed:10000 Mtu:8901} {Name:821b9ce2527df5e MacAddress:a6:ee:ff:79:93:20 Speed:10000 Mtu:8901} {Name:8a19789d7e1d51d MacAddress:f2:33:0a:b1:0c:cd Speed:10000 Mtu:8901} {Name:8eb74a593784fbc MacAddress:ea:77:98:c2:9c:22 Speed:10000 Mtu:8901} {Name:9a302cc5ca4e5b3 MacAddress:26:fd:55:53:e1:12 Speed:10000 Mtu:8901} {Name:b0f5216e5bbf8b4 MacAddress:ea:f4:c2:8c:8e:d6 Speed:10000 Mtu:8901} {Name:b97a30eaa2bd4a1 MacAddress:a6:84:dd:b1:24:05 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:dc:13:c0:0f Speed:0 Mtu:9001} {Name:br-int MacAddress:b6:5d:a2:e9:3c:5b Speed:0 Mtu:8901} {Name:ea73befeff421f1 MacAddress:ea:0a:c2:40:70:ec Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:dc:13:c0:0f Speed:0 Mtu:9001} {Name:f21616762aa00d7 MacAddress:92:6d:93:53:7c:01 Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:92:a0:16:c2:4c:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.525714    4882 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.525780    4882 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526051    4882 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526071    4882 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-252.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526253    4882 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526266    4882 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526279    4882 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526298    4882 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526318    4882 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526618    4882 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526792    4882 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526839    4882 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526856    4882 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526870    4882 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526882    4882 kubelet.go:405] "Adding apiserver pod source"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.526891    4882 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.527786    4882 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.527801    4882 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:42:11.529710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.528295    4882 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 03:42:11.541960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.541937    4882 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 03:42:11.541960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.541957    4882 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:42:11.542118 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542089    4882 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 03:42:11.542361 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542344    4882 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 03:42:11.542502 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542388    4882 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 03:42:11.542579 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542528    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 03:42:11.542579 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542548    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 03:42:11.542579 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542558    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 03:42:11.542579 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542568    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 03:42:11.542579 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542577    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 03:42:11.542795 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542586    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 03:42:11.542795 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542596    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 03:42:11.542795 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542605    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 03:42:11.542795 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542617    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 03:42:11.542795 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542627    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 03:42:11.542795 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542641    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 03:42:11.542795 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542656    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 03:42:11.542795 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542685    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 03:42:11.542795 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.542695    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 03:42:11.547279 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.547255    4882 server.go:1263] "Started kubelet"
Nov 01 03:42:11.548512 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.547601    4882 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 03:42:11.548512 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.547648    4882 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 03:42:11.548512 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.547712    4882 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 03:42:11.548512 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.548111    4882 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 03:42:11.547875 ip-10-0-129-252 systemd[1]: Started Kubernetes Kubelet.
Nov 01 03:42:11.548853 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.548834    4882 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.551038    4882 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.551070    4882 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.551119    4882 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:36:30 +0000 UTC" deadline="2028-05-01 17:18:28.166921517 +0000 UTC"
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.551139    4882 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13141h36m16.615785671s"
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:11.551177    4882 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.551216    4882 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.551228    4882 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.551297    4882 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.553897    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f065d74a-fa1d-41f2-b4a9-6db03bd53a71" volumeName="kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-env-overrides" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.553927    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4070d3e-397e-4202-abb8-ff360ea75ee2" volumeName="kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-snapshots" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.553940    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4070d3e-397e-4202-abb8-ff360ea75ee2" volumeName="kubernetes.io/secret/f4070d3e-397e-4202-abb8-ff360ea75ee2-serving-cert" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.553952    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0271bbb9-50cc-47e2-8681-20e6b2550469" volumeName="kubernetes.io/secret/0271bbb9-50cc-47e2-8681-20e6b2550469-serving-cert" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.553963    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="036c1722-9db5-406e-bc09-0aa6f215e1e0" volumeName="kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-default-certificate" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.553974    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1984d7-7b8d-40a3-b464-ba27be829957" volumeName="kubernetes.io/empty-dir/1d1984d7-7b8d-40a3-b464-ba27be829957-data-volume" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.553987    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42ba278c-e933-4575-872b-3420353c400a" volumeName="kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.553998    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6979e621-5326-4fff-8641-af89a3065735" volumeName="kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-kube-api-access-kqjhm" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554011    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba9103b6-e703-48e0-9276-aec7726fcd36" volumeName="kubernetes.io/projected/ba9103b6-e703-48e0-9276-aec7726fcd36-kube-api-access-gpgbw" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554023    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4070d3e-397e-4202-abb8-ff360ea75ee2" volumeName="kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-service-ca-bundle" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554034    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4070d3e-397e-4202-abb8-ff360ea75ee2" volumeName="kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-tmp" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554045    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cd84ba1-3255-444e-b2e0-2a3d4ac25988" volumeName="kubernetes.io/configmap/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-config" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554056    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd4e00b4-e891-4be2-b941-1238d43e8805" volumeName="kubernetes.io/secret/bd4e00b4-e891-4be2-b941-1238d43e8805-cert" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554067    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4070d3e-397e-4202-abb8-ff360ea75ee2" volumeName="kubernetes.io/projected/f4070d3e-397e-4202-abb8-ff360ea75ee2-kube-api-access-gxjcw" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554081    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f74d5816-8e0a-4f49-83bd-5cd907a57cab" volumeName="kubernetes.io/projected/f74d5816-8e0a-4f49-83bd-5cd907a57cab-kube-api-access-xsxnl" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554092    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0207b2e9-f007-41a7-96c3-aeecde61c873" volumeName="kubernetes.io/configmap/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-cabundle" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554103    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5bf13e-c653-45d9-85f2-a878a8de5236" volumeName="kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-tuned" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554116    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5c8ccd8-df33-4df9-8311-5a163e8e31b3" volumeName="kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-config" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554128    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7c51022-86cd-4cac-bee9-22a33fd53e22" volumeName="kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554139    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a62e18-e602-4b06-ae13-49276b60f80c" volumeName="kubernetes.io/projected/19a62e18-e602-4b06-ae13-49276b60f80c-kube-api-access-2c8lp" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554150    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f065d74a-fa1d-41f2-b4a9-6db03bd53a71" volumeName="kubernetes.io/projected/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-kube-api-access-4fllm" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554160    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0207b2e9-f007-41a7-96c3-aeecde61c873" volumeName="kubernetes.io/secret/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-key" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554183    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="036c1722-9db5-406e-bc09-0aa6f215e1e0" volumeName="kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-stats-auth" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554195    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6979e621-5326-4fff-8641-af89a3065735" volumeName="kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554207    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91524329-4a22-4667-bdeb-fd72ef337912" volumeName="kubernetes.io/projected/91524329-4a22-4667-bdeb-fd72ef337912-kube-api-access-mctgm" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554219    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b496de82-bd77-42c5-8a29-e38514f99c5d" volumeName="kubernetes.io/projected/b496de82-bd77-42c5-8a29-e38514f99c5d-kube-api-access-flxck" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554230    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd1787d7-cd00-470d-b48f-c2d71e566dd2" volumeName="kubernetes.io/projected/cd1787d7-cd00-470d-b48f-c2d71e566dd2-kube-api-access-b42gr" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554240    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f065d74a-fa1d-41f2-b4a9-6db03bd53a71" volumeName="kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-config" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554255    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba9103b6-e703-48e0-9276-aec7726fcd36" volumeName="kubernetes.io/configmap/ba9103b6-e703-48e0-9276-aec7726fcd36-iptables-alerter-script" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554267    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0207b2e9-f007-41a7-96c3-aeecde61c873" volumeName="kubernetes.io/projected/0207b2e9-f007-41a7-96c3-aeecde61c873-kube-api-access-9ds28" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554278    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1984d7-7b8d-40a3-b464-ba27be829957" volumeName="kubernetes.io/projected/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-api-access-8jf8c" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554289    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" volumeName="kubernetes.io/projected/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-kube-api-access-htgqq" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554299    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6979e621-5326-4fff-8641-af89a3065735" volumeName="kubernetes.io/empty-dir/6979e621-5326-4fff-8641-af89a3065735-ca-trust-extracted" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554311    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6979e621-5326-4fff-8641-af89a3065735" volumeName="kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-installation-pull-secrets" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554322    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10" volumeName="kubernetes.io/projected/6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10-kube-api-access-b7vxk" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554332    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5bf13e-c653-45d9-85f2-a878a8de5236" volumeName="kubernetes.io/projected/8d5bf13e-c653-45d9-85f2-a878a8de5236-kube-api-access-bvfj2" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554343    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b07433f-bfec-4e99-97e4-1af1646c45ac" volumeName="kubernetes.io/projected/5b07433f-bfec-4e99-97e4-1af1646c45ac-kube-api-access-wbzqc" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554356    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae127ade-3e32-4aa3-a528-324aa704bcdf" volumeName="kubernetes.io/secret/ae127ade-3e32-4aa3-a528-324aa704bcdf-agent-certs" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554368    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5c8ccd8-df33-4df9-8311-5a163e8e31b3" volumeName="kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-trusted-ca" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554379    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7c51022-86cd-4cac-bee9-22a33fd53e22" volumeName="kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-trusted-ca" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554390    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4070d3e-397e-4202-abb8-ff360ea75ee2" volumeName="kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554401    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f74d5816-8e0a-4f49-83bd-5cd907a57cab" volumeName="kubernetes.io/configmap/f74d5816-8e0a-4f49-83bd-5cd907a57cab-serviceca" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554412    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9f6a2a3-88f9-4d37-b144-371147dd6a6e" volumeName="kubernetes.io/projected/f9f6a2a3-88f9-4d37-b144-371147dd6a6e-kube-api-access-hmxpq" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554423    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20c62809-5ccf-419d-afae-8d140f67309d" volumeName="kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-multus-daemon-config" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554449    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cf6ba1f-8ecb-46fd-a2b9-db4319378237" volumeName="kubernetes.io/configmap/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-nginx-conf" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554460    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c80204c-9fa6-4c25-a1e6-737c7f217976" volumeName="kubernetes.io/projected/9c80204c-9fa6-4c25-a1e6-737c7f217976-kube-api-access-b92hs" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554471    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae127ade-3e32-4aa3-a528-324aa704bcdf" volumeName="kubernetes.io/configmap/ae127ade-3e32-4aa3-a528-324aa704bcdf-konnectivity-ca" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554482    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7c51022-86cd-4cac-bee9-22a33fd53e22" volumeName="kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-installation-pull-secrets" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554494    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" volumeName="kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-binary-copy" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554507    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6979e621-5326-4fff-8641-af89a3065735" volumeName="kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-trusted-ca" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554518    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6979e621-5326-4fff-8641-af89a3065735" volumeName="kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-bound-sa-token" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554529    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d5bf13e-c653-45d9-85f2-a878a8de5236" volumeName="kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-tmp" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554540    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5c8ccd8-df33-4df9-8311-5a163e8e31b3" volumeName="kubernetes.io/secret/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-serving-cert" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554551    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f065d74a-fa1d-41f2-b4a9-6db03bd53a71" volumeName="kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554564    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cd84ba1-3255-444e-b2e0-2a3d4ac25988" volumeName="kubernetes.io/secret/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-serving-cert" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554575    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" volumeName="kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554594    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89e71943-93a0-48b5-aa15-e060cc7b30f3" volumeName="kubernetes.io/projected/89e71943-93a0-48b5-aa15-e060cc7b30f3-kube-api-access-5pxlp" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554605    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91524329-4a22-4667-bdeb-fd72ef337912" volumeName="kubernetes.io/configmap/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-config" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554616    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd4e00b4-e891-4be2-b941-1238d43e8805" volumeName="kubernetes.io/projected/bd4e00b4-e891-4be2-b941-1238d43e8805-kube-api-access-vjqww" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554627    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd24318e-36ed-4fbb-9742-e6e2a9486283" volumeName="kubernetes.io/projected/cd24318e-36ed-4fbb-9742-e6e2a9486283-kube-api-access-z644b" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554640    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d50a0b39-914c-4b62-9b0b-4c6ab787f9d7" volumeName="kubernetes.io/projected/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-kube-api-access-mfd98" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554651    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f065d74a-fa1d-41f2-b4a9-6db03bd53a71" volumeName="kubernetes.io/secret/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554662    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a62e18-e602-4b06-ae13-49276b60f80c" volumeName="kubernetes.io/configmap/19a62e18-e602-4b06-ae13-49276b60f80c-config-volume" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554673    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" volumeName="kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554684    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6979e621-5326-4fff-8641-af89a3065735" volumeName="kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-registry-certificates" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554695    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5c8ccd8-df33-4df9-8311-5a163e8e31b3" volumeName="kubernetes.io/projected/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-kube-api-access-lqvdm" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554707    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7c51022-86cd-4cac-bee9-22a33fd53e22" volumeName="kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-bound-sa-token" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554718    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a62e18-e602-4b06-ae13-49276b60f80c" volumeName="kubernetes.io/empty-dir/19a62e18-e602-4b06-ae13-49276b60f80c-tmp-dir" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554729    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20c62809-5ccf-419d-afae-8d140f67309d" volumeName="kubernetes.io/projected/20c62809-5ccf-419d-afae-8d140f67309d-kube-api-access-g82ht" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554741    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cd84ba1-3255-444e-b2e0-2a3d4ac25988" volumeName="kubernetes.io/projected/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-kube-api-access-zc4z5" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554753    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7c51022-86cd-4cac-bee9-22a33fd53e22" volumeName="kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-certificates" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554764    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d50a0b39-914c-4b62-9b0b-4c6ab787f9d7" volumeName="kubernetes.io/configmap/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-telemetry-config" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554775    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="036c1722-9db5-406e-bc09-0aa6f215e1e0" volumeName="kubernetes.io/projected/036c1722-9db5-406e-bc09-0aa6f215e1e0-kube-api-access-4n9m2" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554786    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c80204c-9fa6-4c25-a1e6-737c7f217976" volumeName="kubernetes.io/empty-dir/9c80204c-9fa6-4c25-a1e6-737c7f217976-tmp-dir" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554797    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7c51022-86cd-4cac-bee9-22a33fd53e22" volumeName="kubernetes.io/empty-dir/c7c51022-86cd-4cac-bee9-22a33fd53e22-ca-trust-extracted" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554810    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0271bbb9-50cc-47e2-8681-20e6b2550469" volumeName="kubernetes.io/configmap/0271bbb9-50cc-47e2-8681-20e6b2550469-config" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554820    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0271bbb9-50cc-47e2-8681-20e6b2550469" volumeName="kubernetes.io/projected/0271bbb9-50cc-47e2-8681-20e6b2550469-kube-api-access-rqmbq" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554831    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20c62809-5ccf-419d-afae-8d140f67309d" volumeName="kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-cni-binary-copy" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554843    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3f80953-074f-455a-9c88-6b3154926050" volumeName="kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554854    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7c51022-86cd-4cac-bee9-22a33fd53e22" volumeName="kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-kube-api-access-msg64" seLinuxMountContext=""
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554865    4882 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.554873    4882 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.555350    4882 factory.go:55] Registering systemd factory
Nov 01 03:42:11.555960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.555369    4882 factory.go:223] Registration of the systemd container factory successfully
Nov 01 03:42:11.571148 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.565955    4882 factory.go:153] Registering CRI-O factory
Nov 01 03:42:11.571148 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.565978    4882 factory.go:223] Registration of the crio container factory successfully
Nov 01 03:42:11.571148 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.566159    4882 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 03:42:11.571148 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.566183    4882 factory.go:103] Registering Raw factory
Nov 01 03:42:11.571148 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.566196    4882 manager.go:1199] Started watching for new ooms in manager
Nov 01 03:42:11.571148 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.567380    4882 manager.go:319] Starting recovery of all containers
Nov 01 03:42:11.576542 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.576521    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:42:11.576639 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.576580    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:42:11.589154 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.589096    4882 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 03:42:11.591677 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.591653    4882 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 03:42:11.591677 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.591676    4882 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 03:42:11.591829 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.591698    4882 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 03:42:11.591829 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:11.591738    4882 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 03:42:11.603744 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.603686    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:42:11.603744 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.603728    4882 generic.go:356] "Generic (PLEG): container finished" podID="b5c8ccd8-df33-4df9-8311-5a163e8e31b3" containerID="2f48c70a28d8e8414666c7e5b90f659b27ace8f85b98bbfcdb4f0f71bdf8242a" exitCode=255
Nov 01 03:42:11.618731 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.618707    4882 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="aeb9a12b8b23e8f2fba77b1e99aa80c221125eb36d21a62e3131e6c19d6d1976" exitCode=0
Nov 01 03:42:11.618731 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.618729    4882 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="41a1ee6742f5cecac1bacbdcad96a512e0ca4c3b011189900e44c7020f4caf27" exitCode=0
Nov 01 03:42:11.618894 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.618737    4882 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="c74679b656c9f86e19a1f18d29b0a85b3febe357962efcaa39c6fe310357709e" exitCode=0
Nov 01 03:42:11.618894 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.618745    4882 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="7720e617a62305528e11d2cc2bd713b4d055a5c8ab954c0716e4e45b79a092cd" exitCode=0
Nov 01 03:42:11.618894 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.618752    4882 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="8d25a9fe6dd518faa4dc04518ca942535d39e86c64720d80eea66e636cc2d9a2" exitCode=0
Nov 01 03:42:11.618894 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.618759    4882 generic.go:356] "Generic (PLEG): container finished" podID="600cdad1-59e2-4bdd-b73e-cb475f7c9cc5" containerID="a13c915b9c1bdc801cade534b923931f0f786769bf29b5917abb6181daff0217" exitCode=0
Nov 01 03:42:11.651413 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:11.651389    4882 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:42:11.665350 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.665322    4882 generic.go:356] "Generic (PLEG): container finished" podID="a5d0b123ba86397e4d3cfebb9bec2811" containerID="b14cff8f77ef0faea1540e809bb47a321741da0e56303fdfd76e620ee64d14fd" exitCode=0
Nov 01 03:42:11.668578 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.668561    4882 manager.go:324] Recovery completed
Nov 01 03:42:11.682150 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.682129    4882 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:42:11.689327 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.689306    4882 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:42:11.690700 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.690677    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:42:11.690769 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.690717    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:42:11.690769 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.690729    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:42:11.692251 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:11.692227    4882 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 03:42:11.692857 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.692840    4882 cpu_manager.go:226] "Starting" policy="none"
Nov 01 03:42:11.692857 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.692855    4882 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 03:42:11.692980 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.692864    4882 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:42:11.692980 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.692876    4882 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 03:42:11.693057 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.693027    4882 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 03:42:11.693093 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.693059    4882 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 03:42:11.693093 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.693083    4882 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 03:42:11.693156 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.693091    4882 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 03:42:11.693156 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.693107    4882 policy_none.go:50] "Start"
Nov 01 03:42:11.693156 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.693122    4882 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 03:42:11.693156 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.693136    4882 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 03:42:11.694058 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.694032    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:42:11.694145 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.694064    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:42:11.694145 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.694078    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:42:11.695415 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.695398    4882 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 03:42:11.695511 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.695419    4882 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 03:42:11.695511 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.695440    4882 policy_none.go:44] "Start"
Nov 01 03:42:11.700369 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.700336    4882 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 03:42:11.703537 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.703521    4882 manager.go:341] "Starting Device Plugin manager"
Nov 01 03:42:11.703628 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:11.703552    4882 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 03:42:11.703628 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.703562    4882 server.go:87] "Starting device plugin registration server"
Nov 01 03:42:11.703876 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.703859    4882 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 03:42:11.704723 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.703880    4882 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 03:42:11.705075 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.705035    4882 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 03:42:11.705194 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.705124    4882 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 03:42:11.705194 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.705171    4882 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 03:42:11.705194 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.705178    4882 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 03:42:11.705342 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.705227    4882 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T03:42:11.705161615Z","UUID":"21afefc2-e667-4abc-a05d-be64c5d77037","Handler":null,"Name":"","Endpoint":""}
Nov 01 03:42:11.706889 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.706863    4882 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 03:42:11.706889 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:11.706879    4882 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 03:42:11.707058 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.706898    4882 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 03:42:11.707058 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:11.706915    4882 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-252.us-west-2.compute.internal\" not found"
Nov 01 03:42:11.725834 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.725811    4882 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:42:11.727397 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.727370    4882 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:42:11.774200 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.774175    4882 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:42:11.774765 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.774741    4882 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 03:42:11.805239 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.805216    4882 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:42:11.806664 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.806643    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:42:11.806787 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.806674    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:42:11.806787 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.806688    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-252.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:42:11.806787 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.806743    4882 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:11.893070 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.892962    4882 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"]
Nov 01 03:42:11.893633 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.893614    4882 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:11.893746 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.893667    4882 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:11.893746 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.893596    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal" event={"ID":"349f3d418ab214cf804b3c489f47ad32","Type":"ContainerStarted","Data":"69308688dc7db7d64687dba04cbe3734d217211b1f39d4c4bb33c4d6afc617b7"}
Nov 01 03:42:11.893860 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.893739    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal" event={"ID":"349f3d418ab214cf804b3c489f47ad32","Type":"ContainerStarted","Data":"a3f7f09bb225b48f6c9a5c9f2adc869de435152a098c04e5c0a484be56f891e5"}
Nov 01 03:42:11.893860 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.893826    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal" event={"ID":"a5d0b123ba86397e4d3cfebb9bec2811","Type":"ContainerStarted","Data":"0587ceef499b0376606eb26e6b4bd8f3d5298295978e30cb7a7d88e22ba6652e"}
Nov 01 03:42:11.893860 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.893847    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal" event={"ID":"a5d0b123ba86397e4d3cfebb9bec2811","Type":"ContainerDied","Data":"b14cff8f77ef0faea1540e809bb47a321741da0e56303fdfd76e620ee64d14fd"}
Nov 01 03:42:11.893976 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.893867    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal" event={"ID":"a5d0b123ba86397e4d3cfebb9bec2811","Type":"ContainerStarted","Data":"c43b6b72bd59316f0febc6194f4dcdd821474d757ab5392f96bd9a800c60abe6"}
Nov 01 03:42:11.917933 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.917905    4882 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:11.918029 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.917961    4882 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:11.951108 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.951090    4882 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:42:11.951208 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:11.951128    4882 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:11.952573 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.952560    4882 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:42:11.952641 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:11.952590    4882 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:11.975709 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.975683    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/349f3d418ab214cf804b3c489f47ad32-config\") pod \"kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"349f3d418ab214cf804b3c489f47ad32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:11.975775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.975711    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:11.975775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:11.975726    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:12.076686 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.076663    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/349f3d418ab214cf804b3c489f47ad32-config\") pod \"kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"349f3d418ab214cf804b3c489f47ad32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:12.076817 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.076690    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:12.076817 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.076707    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:12.076817 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.076746    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:12.076817 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.076770    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/349f3d418ab214cf804b3c489f47ad32-config\") pod \"kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"349f3d418ab214cf804b3c489f47ad32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:12.076817 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.076777    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d0b123ba86397e4d3cfebb9bec2811-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" (UID: \"a5d0b123ba86397e4d3cfebb9bec2811\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:12.527524 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.527492    4882 apiserver.go:52] "Watching apiserver"
Nov 01 03:42:12.669055 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.669024    4882 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:12.669210 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.669187    4882 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:12.670441 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.670414    4882 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 03:42:12.671042 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.671022    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-fwzpl","openshift-network-diagnostics/network-check-target-hbgqw","openshift-insights/insights-runtime-extractor-7rft8","openshift-ovn-kubernetes/ovnkube-node-5tws4","kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal","openshift-multus/multus-m7dq2","openshift-cluster-node-tuning-operator/tuned-kwns5","openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh","openshift-network-console/networking-console-plugin-86c478bb88-2bnwm","openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98","openshift-console-operator/console-operator-7cd56b8dbc-j945n","kube-system/kas-connection-checker-685895c7c9-wgcvj","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md","openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k","kube-system/global-pull-secret-syncer-zdw9t","openshift-image-registry/image-registry-565fcbb6bc-wvnqs","kube-system/kas-connection-checker-685895c7c9-6t6x6","openshift-multus/network-metrics-daemon-vqrc7","openshift-dns/dns-default-7v4p5","openshift-ingress-canary/ingress-canary-46gkp","openshift-ingress/router-default-7dcc875c8f-dlqlr","openshift-insights/insights-operator-7c4986d449-74gj4","openshift-multus/multus-additional-cni-plugins-jz78s","openshift-service-ca/service-ca-5fffc97bdf-6j6bv","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx","openshift-image-registry/image-registry-8894dd498-k59gh","kube-system/kas-connection-checker-685895c7c9-x52jq","kube-system/konnectivity-agent-mm8h5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv","openshift-dns/node-resolver-t8tdn","openshift-network-operator/iptables-alerter-g9vql","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"]
Nov 01 03:42:12.672331 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.672307    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:12.672650 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.672634    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:12.673463 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.673446    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.674795 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.674775    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.675152 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.675136    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:42:12.677281 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.677265    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:12.684141 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.684120    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.685026 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.685009    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:12.685923 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.685905    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.720381 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.720365    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.720527 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.720513    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.721954 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.721936    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 03:42:12.724420 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724395    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.724420 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724416    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.724586 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724559    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.724637 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724592    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 03:42:12.724637 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724604    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 03:42:12.724637 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724625    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-p85rd\""
Nov 01 03:42:12.724781 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724628    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 03:42:12.724781 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724648    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 03:42:12.724781 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724659    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.724781 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724672    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.724917 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724817    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j6wvf\""
Nov 01 03:42:12.724917 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.724831    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 03:42:12.725108 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.725095    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 03:42:12.725335 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.725318    4882 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:42:12.725335 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.725333    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 03:42:12.725441 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.725362    4882 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:12.725547 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.725530    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j4nxv\""
Nov 01 03:42:12.725590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.725571    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 03:42:12.725590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.725585    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 03:42:12.725844 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.725828    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.726013 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.725965    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 03:42:12.726013 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.725986    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.726013 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726008    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.726167 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726143    4882 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:42:12.726225 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726168    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.726225 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726188    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.726225 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726177    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.726225 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.726188    4882 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-252.us-west-2.compute.internal"
Nov 01 03:42:12.726373 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726240    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.726373 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726243    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 03:42:12.726373 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726251    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7gg6p\""
Nov 01 03:42:12.726480 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726451    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.726509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726475    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 03:42:12.726509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726489    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.726509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726499    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 03:42:12.726594 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726503    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.726594 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726538    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 03:42:12.726594 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726547    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.726594 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726550    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-75k7x\""
Nov 01 03:42:12.726594 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726552    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 03:42:12.726928 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.726912    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 03:42:12.753576 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.753560    4882 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 03:42:12.772772 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.772747    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 03:42:12.772857 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.772792    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 03:42:12.773489 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.773467    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 03:42:12.780817 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.780766    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-installation-pull-secrets\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.780817 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.780792    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-installation-pull-secrets\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.780817 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.780813    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-socket-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.780977 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.780829    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.780977 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.780845    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mctgm\" (UniqueName: \"kubernetes.io/projected/91524329-4a22-4667-bdeb-fd72ef337912-kube-api-access-mctgm\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:12.780977 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.780880    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-system-cni-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.780977 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.780909    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.780977 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.780939    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:42:12.781220 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781050    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-installation-pull-secrets\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.781220 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781088    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4z5\" (UniqueName: \"kubernetes.io/projected/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-kube-api-access-zc4z5\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:42:12.781220 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781173    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-installation-pull-secrets\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.781220 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781185    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.781397 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781219    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19a62e18-e602-4b06-ae13-49276b60f80c-config-volume\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:12.781397 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781260    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysconfig\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.781397 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781288    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c80204c-9fa6-4c25-a1e6-737c7f217976-hosts-file\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:42:12.781397 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781326    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42gr\" (UniqueName: \"kubernetes.io/projected/cd1787d7-cd00-470d-b48f-c2d71e566dd2-kube-api-access-b42gr\") pod \"kas-connection-checker-685895c7c9-x52jq\" (UID: \"cd1787d7-cd00-470d-b48f-c2d71e566dd2\") " pod="kube-system/kas-connection-checker-685895c7c9-x52jq"
Nov 01 03:42:12.781397 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781364    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-tmp\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.781662 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781409    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ds28\" (UniqueName: \"kubernetes.io/projected/0207b2e9-f007-41a7-96c3-aeecde61c873-kube-api-access-9ds28\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:12.781662 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781510    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-systemd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.781662 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781543    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fllm\" (UniqueName: \"kubernetes.io/projected/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-kube-api-access-4fllm\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.781662 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781542    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-tmp\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.781662 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781558    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-snapshots\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.781662 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781582    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzqc\" (UniqueName: \"kubernetes.io/projected/5b07433f-bfec-4e99-97e4-1af1646c45ac-kube-api-access-wbzqc\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.781662 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781609    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-stats-auth\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.781662 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781632    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7c51022-86cd-4cac-bee9-22a33fd53e22-ca-trust-extracted\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.781662 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781640    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f4070d3e-397e-4202-abb8-ff360ea75ee2-snapshots\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.781662 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781655    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-bound-sa-token\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781679    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781710    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7c51022-86cd-4cac-bee9-22a33fd53e22-ca-trust-extracted\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781781    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781818    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-trusted-ca\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781846    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-sys\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781873    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae127ade-3e32-4aa3-a528-324aa704bcdf-konnectivity-ca\") pod \"konnectivity-agent-mm8h5\" (UID: \"ae127ade-3e32-4aa3-a528-324aa704bcdf\") " pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781895    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-stats-auth\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781900    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781921    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781949    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-image-registry-private-configuration\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.781972    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782023    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782044    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-trusted-ca\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782061    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82ht\" (UniqueName: \"kubernetes.io/projected/20c62809-5ccf-419d-afae-8d140f67309d-kube-api-access-g82ht\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-certificates\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.782089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782096    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-netns\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782113    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cnibin\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782166    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxpq\" (UniqueName: \"kubernetes.io/projected/f9f6a2a3-88f9-4d37-b144-371147dd6a6e-kube-api-access-hmxpq\") pod \"network-check-source-6f78d59f99-lfx6k\" (UID: \"f9f6a2a3-88f9-4d37-b144-371147dd6a6e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782254    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-trusted-ca\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782310    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-cnibin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782335    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-multus-daemon-config\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782351    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782355    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-trusted-ca\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782375    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f74d5816-8e0a-4f49-83bd-5cd907a57cab-host\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782469    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-systemd\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782495    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a62e18-e602-4b06-ae13-49276b60f80c-tmp-dir\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782547    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-multus-certs\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782576    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-log-socket\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782601    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19a62e18-e602-4b06-ae13-49276b60f80c-tmp-dir\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782606    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxnl\" (UniqueName: \"kubernetes.io/projected/f74d5816-8e0a-4f49-83bd-5cd907a57cab-kube-api-access-xsxnl\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782643    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782707    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba9103b6-e703-48e0-9276-aec7726fcd36-host-slash\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782724    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-bin\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.782766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782762    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-image-registry-private-configuration\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782796    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-var-lib-kubelet\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782848    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvfj2\" (UniqueName: \"kubernetes.io/projected/8d5bf13e-c653-45d9-85f2-a878a8de5236-kube-api-access-bvfj2\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782914    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b92hs\" (UniqueName: \"kubernetes.io/projected/9c80204c-9fa6-4c25-a1e6-737c7f217976-kube-api-access-b92hs\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782973    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.782989    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-kubelet-config\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783005    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-netd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783021    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z644b\" (UniqueName: \"kubernetes.io/projected/cd24318e-36ed-4fbb-9742-e6e2a9486283-kube-api-access-z644b\") pod \"kas-connection-checker-685895c7c9-wgcvj\" (UID: \"cd24318e-36ed-4fbb-9742-e6e2a9486283\") " pod="kube-system/kas-connection-checker-685895c7c9-wgcvj"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783035    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-conf\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783096    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-host\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783195    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-tuned\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783215    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783257    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783266    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba9103b6-e703-48e0-9276-aec7726fcd36-iptables-alerter-script\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783280    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-tuned\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.783329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783318    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-kubelet\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783377    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783414    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-run\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783471    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htgqq\" (UniqueName: \"kubernetes.io/projected/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-kube-api-access-htgqq\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783490    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783506    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae127ade-3e32-4aa3-a528-324aa704bcdf-agent-certs\") pod \"konnectivity-agent-mm8h5\" (UID: \"ae127ade-3e32-4aa3-a528-324aa704bcdf\") " pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783522    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-hostroot\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783568    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-tmp\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783591    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783607    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-sys-fs\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783623    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-system-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783662    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-k8s-cni-cncf-io\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783694    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5bf13e-c653-45d9-85f2-a878a8de5236-tmp\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.783724 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783693    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-default-certificate\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.784086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783754    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-kubelet\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.784086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783779    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0271bbb9-50cc-47e2-8681-20e6b2550469-serving-cert\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:42:12.784086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783793    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f74d5816-8e0a-4f49-83bd-5cd907a57cab-serviceca\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:42:12.784086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783827    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxjcw\" (UniqueName: \"kubernetes.io/projected/f4070d3e-397e-4202-abb8-ff360ea75ee2-kube-api-access-gxjcw\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.784086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783902    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-default-certificate\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.784086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783927    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.784086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783955    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-lib-modules\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.784086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.783978    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-device-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.784086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784032    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msg64\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-kube-api-access-msg64\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.784086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784066    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0271bbb9-50cc-47e2-8681-20e6b2550469-serving-cert\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:42:12.784086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784066    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-registry-certificates\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784102    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-service-ca-bundle\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784219    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8lp\" (UniqueName: \"kubernetes.io/projected/19a62e18-e602-4b06-ae13-49276b60f80c-kube-api-access-2c8lp\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784247    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-node-log\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784272    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovn-node-metrics-cert\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784315    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcbx\" (UniqueName: \"kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx\") pod \"network-check-target-hbgqw\" (UID: \"42ba278c-e933-4575-872b-3420353c400a\") " pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784335    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-netns\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784357    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqvdm\" (UniqueName: \"kubernetes.io/projected/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-kube-api-access-lqvdm\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784373    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-key\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784413    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-bin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784473    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784498    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784519    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-cni-binary-copy\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784542    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-var-lib-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.784590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784580    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-modprobe-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784625    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c80204c-9fa6-4c25-a1e6-737c7f217976-tmp-dir\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784692    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-serving-cert\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784719    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c80204c-9fa6-4c25-a1e6-737c7f217976-tmp-dir\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784756    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-registration-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784758    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4070d3e-397e-4202-abb8-ff360ea75ee2-service-ca-bundle\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784796    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-trusted-ca\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784825    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784854    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784879    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-cabundle\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784906    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9m2\" (UniqueName: \"kubernetes.io/projected/036c1722-9db5-406e-bc09-0aa6f215e1e0-kube-api-access-4n9m2\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784934    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqjhm\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-kube-api-access-kqjhm\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784961    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pxlp\" (UniqueName: \"kubernetes.io/projected/89e71943-93a0-48b5-aa15-e060cc7b30f3-kube-api-access-5pxlp\") pod \"volume-data-source-validator-58c9f5f46d-5t7md\" (UID: \"89e71943-93a0-48b5-aa15-e060cc7b30f3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784989    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-kubernetes\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.784998    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-serving-cert\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785016    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4070d3e-397e-4202-abb8-ff360ea75ee2-serving-cert\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785067    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-socket-dir-parent\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785087    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785103    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785118    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785115    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-trusted-ca\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785159    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f4070d3e-397e-4202-abb8-ff360ea75ee2-serving-cert\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.785228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785209    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-script-lib\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785308    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jf8c\" (UniqueName: \"kubernetes.io/projected/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-api-access-8jf8c\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785328    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-dbus\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785365    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-conf-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785393    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-env-overrides\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785453    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmbq\" (UniqueName: \"kubernetes.io/projected/0271bbb9-50cc-47e2-8681-20e6b2550469-kube-api-access-rqmbq\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785473    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6979e621-5326-4fff-8641-af89a3065735-ca-trust-extracted\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785544    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6979e621-5326-4fff-8641-af89a3065735-ca-trust-extracted\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785586    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d1984d7-7b8d-40a3-b464-ba27be829957-data-volume\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785625    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785650    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd4e00b4-e891-4be2-b941-1238d43e8805-cert\") pod \"ingress-canary-46gkp\" (UID: \"bd4e00b4-e891-4be2-b941-1238d43e8805\") " pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785695    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfd98\" (UniqueName: \"kubernetes.io/projected/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-kube-api-access-mfd98\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785721    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-bound-sa-token\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.785786    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d1984d7-7b8d-40a3-b464-ba27be829957-data-volume\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786014    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786084    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpgbw\" (UniqueName: \"kubernetes.io/projected/ba9103b6-e703-48e0-9276-aec7726fcd36-kube-api-access-gpgbw\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786115    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-etc-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786155    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-config\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786184    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjqww\" (UniqueName: \"kubernetes.io/projected/bd4e00b4-e891-4be2-b941-1238d43e8805-kube-api-access-vjqww\") pod \"ingress-canary-46gkp\" (UID: \"bd4e00b4-e891-4be2-b941-1238d43e8805\") " pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786213    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0271bbb9-50cc-47e2-8681-20e6b2550469-config\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:42:12.786283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786240    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-systemd-units\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786295    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786325    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786354    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-multus\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786383    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vxk\" (UniqueName: \"kubernetes.io/projected/6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10-kube-api-access-b7vxk\") pod \"kas-connection-checker-685895c7c9-6t6x6\" (UID: \"6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10\") " pod="kube-system/kas-connection-checker-685895c7c9-6t6x6"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786408    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-ovn\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786479    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786495    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786528    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-config\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786616    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-etc-kubernetes\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786646    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxck\" (UniqueName: \"kubernetes.io/projected/b496de82-bd77-42c5-8a29-e38514f99c5d-kube-api-access-flxck\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786679    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d1984d7-7b8d-40a3-b464-ba27be829957-crio-socket\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786708    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786735    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-slash\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.786958    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-config\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.787021    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0271bbb9-50cc-47e2-8681-20e6b2550469-config\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:42:12.787236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.787181    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:12.820759 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.820666    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 03:42:12.820850 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.820773    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.820850 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.820791    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.820970 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.820868    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.821043 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821026    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.821096 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821080    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.821143 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821098    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.821203 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821159    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 03:42:12.821203 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821169    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.821203 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821177    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.821203 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821182    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.821399 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821208    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 03:42:12.821399 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821211    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.821399 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821228    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821410    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19a62e18-e602-4b06-ae13-49276b60f80c-config-volume\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821409    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821416    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821468    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821496    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821501    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821511    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821517    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821512    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821531    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821547    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821551    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.821549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821547    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.822169 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821560    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 03:42:12.822169 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821565    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 03:42:12.822169 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821594    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 03:42:12.822169 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821604    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 03:42:12.822169 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821597    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 03:42:12.822169 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821615    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:42:12.822169 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821638    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 03:42:12.822169 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.821700    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 03:42:12.822549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822341    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae127ade-3e32-4aa3-a528-324aa704bcdf-konnectivity-ca\") pod \"konnectivity-agent-mm8h5\" (UID: \"ae127ade-3e32-4aa3-a528-324aa704bcdf\") " pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:42:12.822549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822353    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.822549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822378    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jx8x8\""
Nov 01 03:42:12.822549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822500    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 03:42:12.822734 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822593    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 03:42:12.822734 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822631    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 03:42:12.822734 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822663    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-multus-daemon-config\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.822869 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822742    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 03:42:12.822869 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822788    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-certificates\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.822869 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822830    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jgbfn\""
Nov 01 03:42:12.822869 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822840    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7tm86\""
Nov 01 03:42:12.822869 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.822863    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 03:42:12.823154 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823138    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 03:42:12.823154 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823134    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-image-registry-private-configuration\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823263    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823284    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h4w6m\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823288    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-v2zmn\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823303    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823322    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zn4h5\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823336    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-wjdgr\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823349    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823362    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8fn5p\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823374    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jjmfj\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823377    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823386    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6nkgv\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823374    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jzxr2\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823398    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qv2jj\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823458    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823483    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g6x5b\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823464    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2mnqj\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823487    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tmt47\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823310    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823490    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dc6ls\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823483    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-prgzf\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823515    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823525    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-69w79\""
Nov 01 03:42:12.823564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823519    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v5dfc\""
Nov 01 03:42:12.824187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823584    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba9103b6-e703-48e0-9276-aec7726fcd36-iptables-alerter-script\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:42:12.824187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.823751    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae127ade-3e32-4aa3-a528-324aa704bcdf-agent-certs\") pod \"konnectivity-agent-mm8h5\" (UID: \"ae127ade-3e32-4aa3-a528-324aa704bcdf\") " pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:42:12.824712 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.824688    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovn-node-metrics-cert\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.824804 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.824756    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-key\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:12.824844 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.824810    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3f80953-074f-455a-9c88-6b3154926050-original-pull-secret\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:42:12.824844 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.824815    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f74d5816-8e0a-4f49-83bd-5cd907a57cab-serviceca\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:42:12.824930 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.824851    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20c62809-5ccf-419d-afae-8d140f67309d-cni-binary-copy\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.824930 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.824909    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-registry-certificates\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.825960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.825937    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0207b2e9-f007-41a7-96c3-aeecde61c873-signing-cabundle\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:12.826042 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.825941    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-env-overrides\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.826042 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.825988    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-script-lib\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.826106 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.826038    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd4e00b4-e891-4be2-b941-1238d43e8805-cert\") pod \"ingress-canary-46gkp\" (UID: \"bd4e00b4-e891-4be2-b941-1238d43e8805\") " pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:42:12.827210 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.827197    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-ovnkube-config\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.827261 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.827206    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.832324 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.832309    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-image-registry-private-configuration\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.832383 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.832367    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:42:12.836225 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.836209    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4z5\" (UniqueName: \"kubernetes.io/projected/3cd84ba1-3255-444e-b2e0-2a3d4ac25988-kube-api-access-zc4z5\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6krx\" (UID: \"3cd84ba1-3255-444e-b2e0-2a3d4ac25988\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx"
Nov 01 03:42:12.838200 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.837738    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42gr\" (UniqueName: \"kubernetes.io/projected/cd1787d7-cd00-470d-b48f-c2d71e566dd2-kube-api-access-b42gr\") pod \"kas-connection-checker-685895c7c9-x52jq\" (UID: \"cd1787d7-cd00-470d-b48f-c2d71e566dd2\") " pod="kube-system/kas-connection-checker-685895c7c9-x52jq"
Nov 01 03:42:12.838200 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.837808    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82ht\" (UniqueName: \"kubernetes.io/projected/20c62809-5ccf-419d-afae-8d140f67309d-kube-api-access-g82ht\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.838200 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.837879    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxpq\" (UniqueName: \"kubernetes.io/projected/f9f6a2a3-88f9-4d37-b144-371147dd6a6e-kube-api-access-hmxpq\") pod \"network-check-source-6f78d59f99-lfx6k\" (UID: \"f9f6a2a3-88f9-4d37-b144-371147dd6a6e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k"
Nov 01 03:42:12.838200 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.838085    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z644b\" (UniqueName: \"kubernetes.io/projected/cd24318e-36ed-4fbb-9742-e6e2a9486283-kube-api-access-z644b\") pod \"kas-connection-checker-685895c7c9-wgcvj\" (UID: \"cd24318e-36ed-4fbb-9742-e6e2a9486283\") " pod="kube-system/kas-connection-checker-685895c7c9-wgcvj"
Nov 01 03:42:12.838200 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.838114    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxnl\" (UniqueName: \"kubernetes.io/projected/f74d5816-8e0a-4f49-83bd-5cd907a57cab-kube-api-access-xsxnl\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:42:12.838200 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.838147    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mctgm\" (UniqueName: \"kubernetes.io/projected/91524329-4a22-4667-bdeb-fd72ef337912-kube-api-access-mctgm\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:12.838578 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.838233    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-bound-sa-token\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.838578 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.838323    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b92hs\" (UniqueName: \"kubernetes.io/projected/9c80204c-9fa6-4c25-a1e6-737c7f217976-kube-api-access-b92hs\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:42:12.838578 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.838394    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzqc\" (UniqueName: \"kubernetes.io/projected/5b07433f-bfec-4e99-97e4-1af1646c45ac-kube-api-access-wbzqc\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.838578 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.838480    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvfj2\" (UniqueName: \"kubernetes.io/projected/8d5bf13e-c653-45d9-85f2-a878a8de5236-kube-api-access-bvfj2\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.838855 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.838837    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ds28\" (UniqueName: \"kubernetes.io/projected/0207b2e9-f007-41a7-96c3-aeecde61c873-kube-api-access-9ds28\") pod \"service-ca-5fffc97bdf-6j6bv\" (UID: \"0207b2e9-f007-41a7-96c3-aeecde61c873\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6j6bv"
Nov 01 03:42:12.838912 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.838838    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fllm\" (UniqueName: \"kubernetes.io/projected/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-kube-api-access-4fllm\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.844096 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.844078    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgqq\" (UniqueName: \"kubernetes.io/projected/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-kube-api-access-htgqq\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.844156 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.844135    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxjcw\" (UniqueName: \"kubernetes.io/projected/f4070d3e-397e-4202-abb8-ff360ea75ee2-kube-api-access-gxjcw\") pod \"insights-operator-7c4986d449-74gj4\" (UID: \"f4070d3e-397e-4202-abb8-ff360ea75ee2\") " pod="openshift-insights/insights-operator-7c4986d449-74gj4"
Nov 01 03:42:12.847985 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.847969    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7vxk\" (UniqueName: \"kubernetes.io/projected/6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10-kube-api-access-b7vxk\") pod \"kas-connection-checker-685895c7c9-6t6x6\" (UID: \"6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10\") " pod="kube-system/kas-connection-checker-685895c7c9-6t6x6"
Nov 01 03:42:12.849441 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.849405    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8lp\" (UniqueName: \"kubernetes.io/projected/19a62e18-e602-4b06-ae13-49276b60f80c-kube-api-access-2c8lp\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:12.850410 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.849704    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpgbw\" (UniqueName: \"kubernetes.io/projected/ba9103b6-e703-48e0-9276-aec7726fcd36-kube-api-access-gpgbw\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:42:12.850410 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.849720    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jf8c\" (UniqueName: \"kubernetes.io/projected/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-api-access-8jf8c\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.850410 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.849895    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9m2\" (UniqueName: \"kubernetes.io/projected/036c1722-9db5-406e-bc09-0aa6f215e1e0-kube-api-access-4n9m2\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.850410 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.850196    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjqww\" (UniqueName: \"kubernetes.io/projected/bd4e00b4-e891-4be2-b941-1238d43e8805-kube-api-access-vjqww\") pod \"ingress-canary-46gkp\" (UID: \"bd4e00b4-e891-4be2-b941-1238d43e8805\") " pod="openshift-ingress-canary/ingress-canary-46gkp"
Nov 01 03:42:12.850410 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.850312    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxck\" (UniqueName: \"kubernetes.io/projected/b496de82-bd77-42c5-8a29-e38514f99c5d-kube-api-access-flxck\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:12.850410 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.850367    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msg64\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-kube-api-access-msg64\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.850410 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.850370    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcbx\" (UniqueName: \"kubernetes.io/projected/42ba278c-e933-4575-872b-3420353c400a-kube-api-access-4gcbx\") pod \"network-check-target-hbgqw\" (UID: \"42ba278c-e933-4575-872b-3420353c400a\") " pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:42:12.850410 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.850398    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqvdm\" (UniqueName: \"kubernetes.io/projected/b5c8ccd8-df33-4df9-8311-5a163e8e31b3-kube-api-access-lqvdm\") pod \"console-operator-7cd56b8dbc-j945n\" (UID: \"b5c8ccd8-df33-4df9-8311-5a163e8e31b3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:12.850778 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.850531    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfd98\" (UniqueName: \"kubernetes.io/projected/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-kube-api-access-mfd98\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:12.850778 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.850544    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pxlp\" (UniqueName: \"kubernetes.io/projected/89e71943-93a0-48b5-aa15-e060cc7b30f3-kube-api-access-5pxlp\") pod \"volume-data-source-validator-58c9f5f46d-5t7md\" (UID: \"89e71943-93a0-48b5-aa15-e060cc7b30f3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-5t7md"
Nov 01 03:42:12.850778 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.850642    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-bound-sa-token\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.850778 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.850772    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmbq\" (UniqueName: \"kubernetes.io/projected/0271bbb9-50cc-47e2-8681-20e6b2550469-kube-api-access-rqmbq\") pod \"service-ca-operator-544d749c8c-8rk98\" (UID: \"0271bbb9-50cc-47e2-8681-20e6b2550469\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98"
Nov 01 03:42:12.853375 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.853358    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqjhm\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-kube-api-access-kqjhm\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.887853 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.887826    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:12.887952 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.887862    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-kubernetes\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.887952 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.887886    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-socket-dir-parent\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.887952 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.887912    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.887952 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.887939    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.887958    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-kubernetes\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.887970    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-socket-dir-parent\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.887975    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888015    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-dbus\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888028    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888033    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888039    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-conf-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.888057    4882 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888073    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-multus-conf-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888074    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-dbus\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.888120    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls podName:91524329-4a22-4667-bdeb-fd72ef337912 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:13.388101364 +0000 UTC m=+1.950134366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zvcfn" (UID: "91524329-4a22-4667-bdeb-fd72ef337912") : secret "samples-operator-tls" not found
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.888139    4882 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888138    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-etc-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.888192    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:13.388175339 +0000 UTC m=+1.950208348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:12.888191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888186    4882 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888198    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-etc-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888211    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-systemd-units\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888235    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-systemd-units\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888239    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888268    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-multus\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888298    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-ovn\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888323    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888334    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-multus\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888347    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-etc-kubernetes\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888351    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888384    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-ovn\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888394    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-etc-kubernetes\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.888405    4882 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888417    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d1984d7-7b8d-40a3-b464-ba27be829957-crio-socket\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.888462    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs podName:b496de82-bd77-42c5-8a29-e38514f99c5d nodeName:}" failed. No retries permitted until 2026-11-01 03:42:13.38844974 +0000 UTC m=+1.950482739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs") pod "network-metrics-daemon-vqrc7" (UID: "b496de82-bd77-42c5-8a29-e38514f99c5d") : secret "metrics-daemon-secret" not found
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888467    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d1984d7-7b8d-40a3-b464-ba27be829957-crio-socket\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888480    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888507    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-slash\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888534    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-socket-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888557    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888581    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-system-cni-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888604    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888626    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-socket-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888628    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888628    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888685    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-system-cni-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888685    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysconfig\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.888693    4882 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.888704    4882 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8894dd498-k59gh: secret "image-registry-tls" not found
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888707    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c80204c-9fa6-4c25-a1e6-737c7f217976-hosts-file\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888724    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-systemd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888724    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysconfig\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.888738    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls podName:6979e621-5326-4fff-8641-af89a3065735 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:13.388728319 +0000 UTC m=+1.950761318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls") pod "image-registry-8894dd498-k59gh" (UID: "6979e621-5326-4fff-8641-af89a3065735") : secret "image-registry-tls" not found
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888659    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-slash\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888746    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-run-systemd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.888790    4882 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888794    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c80204c-9fa6-4c25-a1e6-737c7f217976-hosts-file\") pod \"node-resolver-t8tdn\" (UID: \"9c80204c-9fa6-4c25-a1e6-737c7f217976\") " pod="openshift-dns/node-resolver-t8tdn"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888795    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-sys\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888841    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.888856    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:13.388835872 +0000 UTC m=+1.950868881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : configmap "kube-rbac-proxy" not found
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888866    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-sys\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.888877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888887    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-netns\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888910    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cnibin\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888932    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-cnibin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888953    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888973    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f74d5816-8e0a-4f49-83bd-5cd907a57cab-host\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.888996    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-systemd\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889001    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-cnibin\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889019    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-multus-certs\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889043    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-log-socket\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.889053    4882 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.889068    4882 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565fcbb6bc-wvnqs: secret "image-registry-tls" not found
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889068    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889091    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba9103b6-e703-48e0-9276-aec7726fcd36-host-slash\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889113    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-bin\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.889133    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls podName:c7c51022-86cd-4cac-bee9-22a33fd53e22 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:13.389118356 +0000 UTC m=+1.951151363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls") pod "image-registry-565fcbb6bc-wvnqs" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22") : secret "image-registry-tls" not found
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889143    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-log-socket\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889154    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-bin\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889069    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-netns\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889164    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-var-lib-kubelet\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889170    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-etc-selinux\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889113    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-cnibin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889194    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-kubelet-config\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889208    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-multus-certs\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889233    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-netd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889245    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-var-lib-kubelet\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889259    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-conf\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889265    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3f80953-074f-455a-9c88-6b3154926050-kubelet-config\") pod \"global-pull-secret-syncer-zdw9t\" (UID: \"b3f80953-074f-455a-9c88-6b3154926050\") " pod="kube-system/global-pull-secret-syncer-zdw9t"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889281    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-host\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889299    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-cni-netd\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889310    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-kubelet\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889303    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f74d5816-8e0a-4f49-83bd-5cd907a57cab-host\") pod \"node-ca-fwzpl\" (UID: \"f74d5816-8e0a-4f49-83bd-5cd907a57cab\") " pod="openshift-image-registry/node-ca-fwzpl"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889346    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889367    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-host\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889383    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-run\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889407    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-kubelet\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889408    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889424    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-sysctl-conf\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889465    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-hostroot\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889470    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889490    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889513    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-sys-fs\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.889534    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:13.389519281 +0000 UTC m=+1.951552294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : configmap references non-existent config key: service-ca.crt
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889535    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-run\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889560    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-sys-fs\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889581    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba9103b6-e703-48e0-9276-aec7726fcd36-host-slash\") pod \"iptables-alerter-g9vql\" (UID: \"ba9103b6-e703-48e0-9276-aec7726fcd36\") " pod="openshift-network-operator/iptables-alerter-g9vql"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889585    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-hostroot\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889610    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/600cdad1-59e2-4bdd-b73e-cb475f7c9cc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jz78s\" (UID: \"600cdad1-59e2-4bdd-b73e-cb475f7c9cc5\") " pod="openshift-multus/multus-additional-cni-plugins-jz78s"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889638    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-systemd\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889720    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-system-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.889564    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-system-cni-dir\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890033    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-k8s-cni-cncf-io\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890060    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-kubelet\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890086    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890110    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-lib-modules\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890131    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-k8s-cni-cncf-io\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890134    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-device-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890140    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-kubelet\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890175    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-node-log\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890187    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-device-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890203    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-netns\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890230    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-bin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890250    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-node-log\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890286    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-lib-modules\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890292    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-var-lib-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890202    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890322    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-var-lib-cni-bin\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890289    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20c62809-5ccf-419d-afae-8d140f67309d-host-run-netns\") pod \"multus-m7dq2\" (UID: \"20c62809-5ccf-419d-afae-8d140f67309d\") " pod="openshift-multus/multus-m7dq2"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890255    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f065d74a-fa1d-41f2-b4a9-6db03bd53a71-var-lib-openvswitch\") pod \"ovnkube-node-5tws4\" (UID: \"f065d74a-fa1d-41f2-b4a9-6db03bd53a71\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890358    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-modprobe-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890385    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-registration-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890409    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.890532    4882 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:12.890558 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:12.890583    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls podName:d50a0b39-914c-4b62-9b0b-4c6ab787f9d7 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:13.390569089 +0000 UTC m=+1.952602101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pknwh" (UID: "d50a0b39-914c-4b62-9b0b-4c6ab787f9d7") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:12.892719 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890674    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5bf13e-c653-45d9-85f2-a878a8de5236-etc-modprobe-d\") pod \"tuned-kwns5\" (UID: \"8d5bf13e-c653-45d9-85f2-a878a8de5236\") " pod="openshift-cluster-node-tuning-operator/tuned-kwns5"
Nov 01 03:42:12.892719 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.890720    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b07433f-bfec-4e99-97e4-1af1646c45ac-registration-dir\") pod \"aws-ebs-csi-driver-node-wwzcv\" (UID: \"5b07433f-bfec-4e99-97e4-1af1646c45ac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wwzcv"
Nov 01 03:42:12.892719 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.891047    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19a62e18-e602-4b06-ae13-49276b60f80c-metrics-tls\") pod \"dns-default-7v4p5\" (UID: \"19a62e18-e602-4b06-ae13-49276b60f80c\") " pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:12.892719 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.891279    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cf6ba1f-8ecb-46fd-a2b9-db4319378237-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2bnwm\" (UID: \"5cf6ba1f-8ecb-46fd-a2b9-db4319378237\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:42:12.892719 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.891729    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/036c1722-9db5-406e-bc09-0aa6f215e1e0-metrics-certs\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:12.973958 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.973936    4882 scope.go:122] "RemoveContainer" containerID="2f48c70a28d8e8414666c7e5b90f659b27ace8f85b98bbfcdb4f0f71bdf8242a"
Nov 01 03:42:12.975173 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.975158    4882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:42:12.975772 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.975757    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"
Nov 01 03:42:12.986251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:12.986229    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:13.370470 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.370380    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2bnwm"]
Nov 01 03:42:13.403817 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.403790    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:13.403983 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.403838    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:13.403983 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.403869    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:13.403983 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.403962    4882 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:42:13.403983 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.403967    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:14.403949587 +0000 UTC m=+2.965982600 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : configmap references non-existent config key: service-ca.crt
Nov 01 03:42:13.403983 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.403973    4882 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:13.404222 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.403999    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls podName:91524329-4a22-4667-bdeb-fd72ef337912 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:14.403988276 +0000 UTC m=+2.966021272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zvcfn" (UID: "91524329-4a22-4667-bdeb-fd72ef337912") : secret "samples-operator-tls" not found
Nov 01 03:42:13.404222 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.404027    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls podName:d50a0b39-914c-4b62-9b0b-4c6ab787f9d7 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:14.404014988 +0000 UTC m=+2.966048002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pknwh" (UID: "d50a0b39-914c-4b62-9b0b-4c6ab787f9d7") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:13.404222 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.404046    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:13.404222 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.404083    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:13.404222 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.404111    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:13.404222 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.404129    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:13.404222 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.404152    4882 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:13.404222 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.404164    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:13.404222 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.404206    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:14.404190751 +0000 UTC m=+2.966223760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:13.404222 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.404161    4882 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:42:13.404693 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.404249    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs podName:b496de82-bd77-42c5-8a29-e38514f99c5d nodeName:}" failed. No retries permitted until 2026-11-01 03:42:14.404239637 +0000 UTC m=+2.966272635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs") pod "network-metrics-daemon-vqrc7" (UID: "b496de82-bd77-42c5-8a29-e38514f99c5d") : secret "metrics-daemon-secret" not found
Nov 01 03:42:13.404693 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.404249    4882 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:13.404693 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.404259    4882 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:13.404693 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.404264    4882 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8894dd498-k59gh: secret "image-registry-tls" not found
Nov 01 03:42:13.404693 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.404268    4882 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565fcbb6bc-wvnqs: secret "image-registry-tls" not found
Nov 01 03:42:13.404693 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.404294    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls podName:c7c51022-86cd-4cac-bee9-22a33fd53e22 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:14.40428562 +0000 UTC m=+2.966318628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls") pod "image-registry-565fcbb6bc-wvnqs" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22") : secret "image-registry-tls" not found
Nov 01 03:42:13.404693 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:13.404307    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls podName:6979e621-5326-4fff-8641-af89a3065735 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:14.404301182 +0000 UTC m=+2.966334176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls") pod "image-registry-8894dd498-k59gh" (UID: "6979e621-5326-4fff-8641-af89a3065735") : secret "image-registry-tls" not found
Nov 01 03:42:13.404693 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.404617    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d1984d7-7b8d-40a3-b464-ba27be829957-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:13.420797 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:13.420760    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf6ba1f_8ecb_46fd_a2b9_db4319378237.slice/crio-1a354b315a69461b4b60178c9220b2a4056a8b0b1bb073571971922933ddf148 WatchSource:0}: Error finding container 1a354b315a69461b4b60178c9220b2a4056a8b0b1bb073571971922933ddf148: Status 404 returned error can't find the container with id 1a354b315a69461b4b60178c9220b2a4056a8b0b1bb073571971922933ddf148
Nov 01 03:42:13.424331 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.424313    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7v4p5"]
Nov 01 03:42:13.444976 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:13.444955    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a62e18_e602_4b06_ae13_49276b60f80c.slice/crio-2abe4531729d1efb3fe4ab74077ae52cc14b9de9479fbb3f7eadddbf827fb6cc WatchSource:0}: Error finding container 2abe4531729d1efb3fe4ab74077ae52cc14b9de9479fbb3f7eadddbf827fb6cc: Status 404 returned error can't find the container with id 2abe4531729d1efb3fe4ab74077ae52cc14b9de9479fbb3f7eadddbf827fb6cc
Nov 01 03:42:13.673949 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.673875    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7v4p5" event={"ID":"19a62e18-e602-4b06-ae13-49276b60f80c","Type":"ContainerStarted","Data":"2abe4531729d1efb3fe4ab74077ae52cc14b9de9479fbb3f7eadddbf827fb6cc"}
Nov 01 03:42:13.675462 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.675415    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lfx6k" event={"ID":"f9f6a2a3-88f9-4d37-b144-371147dd6a6e","Type":"ContainerStarted","Data":"26d871d4ce6f48e3be14e060cf4a729add0470ebe5dc55a9de1d5fe668f47ba3"}
Nov 01 03:42:13.676604 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.676576    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm" event={"ID":"5cf6ba1f-8ecb-46fd-a2b9-db4319378237","Type":"ContainerStarted","Data":"1a354b315a69461b4b60178c9220b2a4056a8b0b1bb073571971922933ddf148"}
Nov 01 03:42:13.678414 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.678394    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:42:13.678534 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:13.678515    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n" event={"ID":"b5c8ccd8-df33-4df9-8311-5a163e8e31b3","Type":"ContainerStarted","Data":"d796db6e77d371d055f81002a7f0e9fcbca7fcabc1fe1dd5e831e944b50ca0f8"}
Nov 01 03:42:14.333001 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:14.332973    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:14.413152 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:14.413070    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:14.413152 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:14.413110    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:14.413355 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413199    4882 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:14.413355 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413205    4882 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:42:14.413355 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:14.413209    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:14.413355 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:14.413265    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:14.413355 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413211    4882 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8894dd498-k59gh: secret "image-registry-tls" not found
Nov 01 03:42:14.413355 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413284    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs podName:b496de82-bd77-42c5-8a29-e38514f99c5d nodeName:}" failed. No retries permitted until 2026-11-01 03:42:16.413264926 +0000 UTC m=+4.975297948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs") pod "network-metrics-daemon-vqrc7" (UID: "b496de82-bd77-42c5-8a29-e38514f99c5d") : secret "metrics-daemon-secret" not found
Nov 01 03:42:14.413355 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413354    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls podName:6979e621-5326-4fff-8641-af89a3065735 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:16.41334022 +0000 UTC m=+4.975373228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls") pod "image-registry-8894dd498-k59gh" (UID: "6979e621-5326-4fff-8641-af89a3065735") : secret "image-registry-tls" not found
Nov 01 03:42:14.413699 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413419    4882 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:14.413699 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413447    4882 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565fcbb6bc-wvnqs: secret "image-registry-tls" not found
Nov 01 03:42:14.413699 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:14.413455    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:14.413699 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413472    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:16.413452937 +0000 UTC m=+4.975485953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : configmap references non-existent config key: service-ca.crt
Nov 01 03:42:14.413699 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413507    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls podName:c7c51022-86cd-4cac-bee9-22a33fd53e22 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:16.41349699 +0000 UTC m=+4.975529988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls") pod "image-registry-565fcbb6bc-wvnqs" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22") : secret "image-registry-tls" not found
Nov 01 03:42:14.413699 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413527    4882 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:14.413699 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:14.413545    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:14.413699 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413564    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls podName:d50a0b39-914c-4b62-9b0b-4c6ab787f9d7 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:16.413552187 +0000 UTC m=+4.975585182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pknwh" (UID: "d50a0b39-914c-4b62-9b0b-4c6ab787f9d7") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:14.413699 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:14.413632    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:14.413699 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413671    4882 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:42:14.413998 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413714    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls podName:91524329-4a22-4667-bdeb-fd72ef337912 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:16.413700975 +0000 UTC m=+4.975733970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-zvcfn" (UID: "91524329-4a22-4667-bdeb-fd72ef337912") : secret "samples-operator-tls" not found
Nov 01 03:42:14.413998 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413724    4882 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:14.413998 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:14.413769    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:16.41374863 +0000 UTC m=+4.975781645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:14.683686 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:14.683606    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx" event={"ID":"3cd84ba1-3255-444e-b2e0-2a3d4ac25988","Type":"ContainerStarted","Data":"e90a70f5be20f53fc184aa0be06e230dc95053c8186a4142eab62e79c5cfab38"}
Nov 01 03:42:15.689403 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:15.689370    4882 generic.go:356] "Generic (PLEG): container finished" podID="0271bbb9-50cc-47e2-8681-20e6b2550469" containerID="76a16282a6954301d080e5d0b1d8462ce1f7468cc247f1a4295e9ac66d31c5e4" exitCode=0
Nov 01 03:42:15.689812 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:15.689454    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98" event={"ID":"0271bbb9-50cc-47e2-8681-20e6b2550469","Type":"ContainerDied","Data":"76a16282a6954301d080e5d0b1d8462ce1f7468cc247f1a4295e9ac66d31c5e4"}
Nov 01 03:42:15.689812 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:15.689749    4882 scope.go:122] "RemoveContainer" containerID="76a16282a6954301d080e5d0b1d8462ce1f7468cc247f1a4295e9ac66d31c5e4"
Nov 01 03:42:16.432637 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.432597    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:16.432637 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.432639    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:16.432905 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.432683    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:16.432905 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.432707    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:16.432905 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.432735    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:16.432905 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.432817    4882 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:16.432905 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.432825    4882 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:42:16.432905 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.432852    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:16.432905 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.432865    4882 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:16.432905 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.432897    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs podName:b496de82-bd77-42c5-8a29-e38514f99c5d nodeName:}" failed. No retries permitted until 2026-11-01 03:42:20.432875976 +0000 UTC m=+8.994908988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs") pod "network-metrics-daemon-vqrc7" (UID: "b496de82-bd77-42c5-8a29-e38514f99c5d") : secret "metrics-daemon-secret" not found
Nov 01 03:42:16.432905 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.432825    4882 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:42:16.432905 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.432914    4882 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8894dd498-k59gh: secret "image-registry-tls" not found
Nov 01 03:42:16.433332 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.432832    4882 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565fcbb6bc-wvnqs: secret "image-registry-tls" not found
Nov 01 03:42:16.433332 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.432961    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls podName:1d1984d7-7b8d-40a3-b464-ba27be829957 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:20.432929108 +0000 UTC m=+8.994962109 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rft8" (UID: "1d1984d7-7b8d-40a3-b464-ba27be829957") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:42:16.433332 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.432997    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls podName:6979e621-5326-4fff-8641-af89a3065735 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:20.432986042 +0000 UTC m=+8.995019037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls") pod "image-registry-8894dd498-k59gh" (UID: "6979e621-5326-4fff-8641-af89a3065735") : secret "image-registry-tls" not found
Nov 01 03:42:16.433332 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.433021    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle podName:036c1722-9db5-406e-bc09-0aa6f215e1e0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:20.43301228 +0000 UTC m=+8.995045281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle") pod "router-default-7dcc875c8f-dlqlr" (UID: "036c1722-9db5-406e-bc09-0aa6f215e1e0") : configmap references non-existent config key: service-ca.crt
Nov 01 03:42:16.433332 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.433038    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls podName:c7c51022-86cd-4cac-bee9-22a33fd53e22 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:20.433033065 +0000 UTC m=+8.995066060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls") pod "image-registry-565fcbb6bc-wvnqs" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22") : secret "image-registry-tls" not found
Nov 01 03:42:16.433332 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.433086    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:16.433332 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.433197    4882 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:16.433332 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:16.433228    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls podName:d50a0b39-914c-4b62-9b0b-4c6ab787f9d7 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:20.433221212 +0000 UTC m=+8.995254212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pknwh" (UID: "d50a0b39-914c-4b62-9b0b-4c6ab787f9d7") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:42:16.435048 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.435027    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91524329-4a22-4667-bdeb-fd72ef337912-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-zvcfn\" (UID: \"91524329-4a22-4667-bdeb-fd72ef337912\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:16.573284 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.573251    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"
Nov 01 03:42:16.695588 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.695561    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-8rk98" event={"ID":"0271bbb9-50cc-47e2-8681-20e6b2550469","Type":"ContainerStarted","Data":"f782275b5142f06bd57da4386ecb8b52188b92529d2e69fde3011117cbfc3795"}
Nov 01 03:42:16.697182 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.697159    4882 generic.go:356] "Generic (PLEG): container finished" podID="f4070d3e-397e-4202-abb8-ff360ea75ee2" containerID="c896df2f02a0172275eb17f57f4a00aab9422a8fd034343d8be28d92a13fae1c" exitCode=0
Nov 01 03:42:16.697291 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.697227    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-74gj4" event={"ID":"f4070d3e-397e-4202-abb8-ff360ea75ee2","Type":"ContainerDied","Data":"c896df2f02a0172275eb17f57f4a00aab9422a8fd034343d8be28d92a13fae1c"}
Nov 01 03:42:16.697552 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.697539    4882 scope.go:122] "RemoveContainer" containerID="c896df2f02a0172275eb17f57f4a00aab9422a8fd034343d8be28d92a13fae1c"
Nov 01 03:42:16.959487 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:16.959460    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn"]
Nov 01 03:42:17.701096 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:17.701059    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn" event={"ID":"91524329-4a22-4667-bdeb-fd72ef337912","Type":"ContainerStarted","Data":"1202573152bb01a8d96d450aa29fc6d206e2544fcbcb423b222791e8399f4675"}
Nov 01 03:42:17.702947 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:17.702924    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-74gj4" event={"ID":"f4070d3e-397e-4202-abb8-ff360ea75ee2","Type":"ContainerStarted","Data":"2f3b78ba32591659880dcb68a9ef476fab3db7538bcdfa907177d098e48743e3"}
Nov 01 03:42:17.704671 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:17.704638    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2bnwm" event={"ID":"5cf6ba1f-8ecb-46fd-a2b9-db4319378237","Type":"ContainerStarted","Data":"9dc81f5b35877f7e6a31ebc07bc306357861ee1654e1dbc04706894e25276497"}
Nov 01 03:42:19.409360 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:19.409327    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:42:19.584958 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:19.584927    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:19.585193 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:19.585169    4882 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:42:20.472925 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.472888    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:20.472925 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.472930    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:20.473473 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.472953    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:20.473473 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.472981    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:20.473473 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.473008    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:20.473473 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.473040    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:20.474327 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.474190    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/036c1722-9db5-406e-bc09-0aa6f215e1e0-service-ca-bundle\") pod \"router-default-7dcc875c8f-dlqlr\" (UID: \"036c1722-9db5-406e-bc09-0aa6f215e1e0\") " pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:20.476570 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.476542    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"image-registry-565fcbb6bc-wvnqs\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") " pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:20.476689 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.476545    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d50a0b39-914c-4b62-9b0b-4c6ab787f9d7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pknwh\" (UID: \"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:20.476689 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.476614    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"image-registry-8894dd498-k59gh\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") " pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:20.477737 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.477713    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d1984d7-7b8d-40a3-b464-ba27be829957-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rft8\" (UID: \"1d1984d7-7b8d-40a3-b464-ba27be829957\") " pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:20.479962 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.479942    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b496de82-bd77-42c5-8a29-e38514f99c5d-metrics-certs\") pod \"network-metrics-daemon-vqrc7\" (UID: \"b496de82-bd77-42c5-8a29-e38514f99c5d\") " pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:20.484998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.484976    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:20.487115 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.487096    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7rft8"
Nov 01 03:42:20.720391 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.720357    4882 generic.go:356] "Generic (PLEG): container finished" podID="3cd84ba1-3255-444e-b2e0-2a3d4ac25988" containerID="e90a70f5be20f53fc184aa0be06e230dc95053c8186a4142eab62e79c5cfab38" exitCode=0
Nov 01 03:42:20.720567 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.720449    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx" event={"ID":"3cd84ba1-3255-444e-b2e0-2a3d4ac25988","Type":"ContainerDied","Data":"e90a70f5be20f53fc184aa0be06e230dc95053c8186a4142eab62e79c5cfab38"}
Nov 01 03:42:20.720804 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.720791    4882 scope.go:122] "RemoveContainer" containerID="e90a70f5be20f53fc184aa0be06e230dc95053c8186a4142eab62e79c5cfab38"
Nov 01 03:42:20.773351 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.773331    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"
Nov 01 03:42:20.774539 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.774508    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:20.775557 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.775538    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:20.778125 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:20.778104    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqrc7"
Nov 01 03:42:21.006188 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.006116    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7rft8"]
Nov 01 03:42:21.006926 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.006863    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-565fcbb6bc-wvnqs"]
Nov 01 03:42:21.015388 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:21.015365    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c51022_86cd_4cac_bee9_22a33fd53e22.slice/crio-2a58f9d22e46b035d548ea7622a7cba7a9d294128cde5f23e40f170e3f454999 WatchSource:0}: Error finding container 2a58f9d22e46b035d548ea7622a7cba7a9d294128cde5f23e40f170e3f454999: Status 404 returned error can't find the container with id 2a58f9d22e46b035d548ea7622a7cba7a9d294128cde5f23e40f170e3f454999
Nov 01 03:42:21.015888 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:21.015866    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1984d7_7b8d_40a3_b464_ba27be829957.slice/crio-4690f692d91d38f921b4f5122666c16eb96aa16d296a0898aa0077cee7e83e2c WatchSource:0}: Error finding container 4690f692d91d38f921b4f5122666c16eb96aa16d296a0898aa0077cee7e83e2c: Status 404 returned error can't find the container with id 4690f692d91d38f921b4f5122666c16eb96aa16d296a0898aa0077cee7e83e2c
Nov 01 03:42:21.175542 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.175511    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh"]
Nov 01 03:42:21.179146 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.179121    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7dcc875c8f-dlqlr"]
Nov 01 03:42:21.185306 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:21.185281    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50a0b39_914c_4b62_9b0b_4c6ab787f9d7.slice/crio-3dec273311bc07001f12f560b400ba009ad81c4cfe8116850388882688d1a3c0 WatchSource:0}: Error finding container 3dec273311bc07001f12f560b400ba009ad81c4cfe8116850388882688d1a3c0: Status 404 returned error can't find the container with id 3dec273311bc07001f12f560b400ba009ad81c4cfe8116850388882688d1a3c0
Nov 01 03:42:21.188050 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:21.188026    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036c1722_9db5_406e_bc09_0aa6f215e1e0.slice/crio-5112238fefaf1cb8bc9f311657fed5126cb05ac85163cb835d79f1552f70e288 WatchSource:0}: Error finding container 5112238fefaf1cb8bc9f311657fed5126cb05ac85163cb835d79f1552f70e288: Status 404 returned error can't find the container with id 5112238fefaf1cb8bc9f311657fed5126cb05ac85163cb835d79f1552f70e288
Nov 01 03:42:21.226904 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.226877    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vqrc7"]
Nov 01 03:42:21.229840 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.229819    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8894dd498-k59gh"]
Nov 01 03:42:21.236222 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:21.236192    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb496de82_bd77_42c5_8a29_e38514f99c5d.slice/crio-a63efc2029d60d7702da2d61abe5d117541a298f098e2034f8c3997a3ea661d2 WatchSource:0}: Error finding container a63efc2029d60d7702da2d61abe5d117541a298f098e2034f8c3997a3ea661d2: Status 404 returned error can't find the container with id a63efc2029d60d7702da2d61abe5d117541a298f098e2034f8c3997a3ea661d2
Nov 01 03:42:21.239113 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:21.239089    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6979e621_5326_4fff_8641_af89a3065735.slice/crio-f786eae1a5e2176c06131cd2315a87595f105deeb22989148e3e0e82581e627b WatchSource:0}: Error finding container f786eae1a5e2176c06131cd2315a87595f105deeb22989148e3e0e82581e627b: Status 404 returned error can't find the container with id f786eae1a5e2176c06131cd2315a87595f105deeb22989148e3e0e82581e627b
Nov 01 03:42:21.725958 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.725918    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqrc7" event={"ID":"b496de82-bd77-42c5-8a29-e38514f99c5d","Type":"ContainerStarted","Data":"a63efc2029d60d7702da2d61abe5d117541a298f098e2034f8c3997a3ea661d2"}
Nov 01 03:42:21.727118 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.727078    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh" event={"ID":"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7","Type":"ContainerStarted","Data":"3dec273311bc07001f12f560b400ba009ad81c4cfe8116850388882688d1a3c0"}
Nov 01 03:42:21.728727 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.728700    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs" event={"ID":"c7c51022-86cd-4cac-bee9-22a33fd53e22","Type":"ContainerStarted","Data":"bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a"}
Nov 01 03:42:21.728839 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.728753    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs" event={"ID":"c7c51022-86cd-4cac-bee9-22a33fd53e22","Type":"ContainerStarted","Data":"2a58f9d22e46b035d548ea7622a7cba7a9d294128cde5f23e40f170e3f454999"}
Nov 01 03:42:21.730444 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.730405    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8894dd498-k59gh" event={"ID":"6979e621-5326-4fff-8641-af89a3065735","Type":"ContainerStarted","Data":"fd4dd5d559c5421f9c558819e7e4ca68dbb33dadd9612dc1c03cb35ec86dbf7a"}
Nov 01 03:42:21.730559 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.730476    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8894dd498-k59gh" event={"ID":"6979e621-5326-4fff-8641-af89a3065735","Type":"ContainerStarted","Data":"f786eae1a5e2176c06131cd2315a87595f105deeb22989148e3e0e82581e627b"}
Nov 01 03:42:21.733013 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.732862    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rft8" event={"ID":"1d1984d7-7b8d-40a3-b464-ba27be829957","Type":"ContainerStarted","Data":"9ad7b7fa9ffb32a66b42404fed85b1eb141e19390ccfe63584772cfe34c3ec84"}
Nov 01 03:42:21.733013 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.732910    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rft8" event={"ID":"1d1984d7-7b8d-40a3-b464-ba27be829957","Type":"ContainerStarted","Data":"4690f692d91d38f921b4f5122666c16eb96aa16d296a0898aa0077cee7e83e2c"}
Nov 01 03:42:21.733013 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.732984    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:21.735883 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.735813    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7dcc875c8f-dlqlr" event={"ID":"036c1722-9db5-406e-bc09-0aa6f215e1e0","Type":"ContainerStarted","Data":"29a36f49d55814fe31c759498dd0ce1612bed7dc769b6b52287dd8dc5eac5e9d"}
Nov 01 03:42:21.735883 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.735855    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7dcc875c8f-dlqlr" event={"ID":"036c1722-9db5-406e-bc09-0aa6f215e1e0","Type":"ContainerStarted","Data":"5112238fefaf1cb8bc9f311657fed5126cb05ac85163cb835d79f1552f70e288"}
Nov 01 03:42:21.739095 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.739063    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6krx" event={"ID":"3cd84ba1-3255-444e-b2e0-2a3d4ac25988","Type":"ContainerStarted","Data":"0b7a1d13a38f3e61f99c912f088aafe5b557d1f4dd303455f061f2bbc236f474"}
Nov 01 03:42:21.775002 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.774975    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:21.777646 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:21.777626    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:22.002142 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:22.002069    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:22.002300 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:22.002214    4882 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:42:22.006711 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:22.006272    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-j945n"
Nov 01 03:42:22.745453 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:22.745351    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7v4p5" event={"ID":"19a62e18-e602-4b06-ae13-49276b60f80c","Type":"ContainerStarted","Data":"19c5da82866a6c1bb40a927e0965f8afd3d33ff6d29221fc1da04da50218a5c3"}
Nov 01 03:42:22.745453 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:22.745420    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7v4p5" event={"ID":"19a62e18-e602-4b06-ae13-49276b60f80c","Type":"ContainerStarted","Data":"085862bd6e2cd98f130386e688282e0d3de57ae986506a0a258dd8e85cb4ef96"}
Nov 01 03:42:22.745940 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:22.745486    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:22.748495 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:22.748466    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rft8" event={"ID":"1d1984d7-7b8d-40a3-b464-ba27be829957","Type":"ContainerStarted","Data":"33ee70815374775120b8e983afd30611b05710858aaba9ea91739688453e9a54"}
Nov 01 03:42:22.748621 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:22.748555    4882 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:42:23.753772 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:23.753727    4882 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:42:24.759449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:24.759391    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn" event={"ID":"91524329-4a22-4667-bdeb-fd72ef337912","Type":"ContainerStarted","Data":"c091df299f72d6644113e5197bc13202a3640fba0e95287978f46088c3b49831"}
Nov 01 03:42:24.759865 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:24.759510    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-zvcfn" event={"ID":"91524329-4a22-4667-bdeb-fd72ef337912","Type":"ContainerStarted","Data":"b5be08248990388ee36b442d3787b0a786dc41316c7a32069430acb2b85adf8e"}
Nov 01 03:42:24.761180 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:24.761158    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqrc7" event={"ID":"b496de82-bd77-42c5-8a29-e38514f99c5d","Type":"ContainerStarted","Data":"4e14ce16a2147306d1cce20bff8805205b513a3344636346c1650171446bcffe"}
Nov 01 03:42:24.761268 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:24.761199    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqrc7" event={"ID":"b496de82-bd77-42c5-8a29-e38514f99c5d","Type":"ContainerStarted","Data":"eda42adb0b11b1d052002d8ccd13120dbcfca7f58900c0d0eda92b4a132d7ad5"}
Nov 01 03:42:25.250524 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:25.250492    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mm8h5"
Nov 01 03:42:25.766380 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:25.766346    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pknwh" event={"ID":"d50a0b39-914c-4b62-9b0b-4c6ab787f9d7","Type":"ContainerStarted","Data":"20812148dd52df49de92b93e6a8b685807e9b0930460bf14feb0b29d67d843e3"}
Nov 01 03:42:26.395181 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.395145    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-g9bvp-mhv7w"]
Nov 01 03:42:26.399332 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.399310    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-gj465-dgxzq"]
Nov 01 03:42:26.399466 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.399402    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.402669 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.402650    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.441602 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.441568    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-99c76"]
Nov 01 03:42:26.445307 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.445285    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fffc4755-s9qkn"]
Nov 01 03:42:26.445470 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.445413    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-99c76"
Nov 01 03:42:26.448873 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.448850    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.487680 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.487651    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8894dd498-k59gh"]
Nov 01 03:42:26.487789 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.487774    4882 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:42:26.489981 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.489951    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-g9bvp-mhv7w"]
Nov 01 03:42:26.490771 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.490752    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gj465-dgxzq"]
Nov 01 03:42:26.496746 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.496725    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 03:42:26.496952 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.496935    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 03:42:26.513469 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.513418    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r7v4n"]
Nov 01 03:42:26.515507 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.515422    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 01 03:42:26.515507 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.515499    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 01 03:42:26.515638 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.515530    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 01 03:42:26.516729 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.516710    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r7v4n"
Nov 01 03:42:26.524115 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.524093    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b96d2df2-07b5-46db-823b-5dded530efe3-archives-path\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.524215 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.524160    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-service-ca-bundle\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.524293 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.524269    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-archives-path\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.524328 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.524308    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhkfm\" (UniqueName: \"kubernetes.io/projected/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-kube-api-access-rhkfm\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.524539 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.524515    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-trusted-ca-bundle\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.524725 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.524691    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-trusted-ca-bundle\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.524793 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.524741    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-service-ca-bundle\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.524793 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.524767    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llfg\" (UniqueName: \"kubernetes.io/projected/b96d2df2-07b5-46db-823b-5dded530efe3-kube-api-access-7llfg\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.533890 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.533870    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cf89cb94b-cpg2r"]
Nov 01 03:42:26.536971 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.536955    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-twwg4"]
Nov 01 03:42:26.537079 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.537064    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.541258 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.540530    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twwg4"
Nov 01 03:42:26.541258 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.540658    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-99c76"]
Nov 01 03:42:26.541420 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.541346    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fffc4755-s9qkn"]
Nov 01 03:42:26.570392 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.570370    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 03:42:26.570514 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.570497    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 03:42:26.577994 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.577972    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r7v4n"]
Nov 01 03:42:26.581798 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.581620    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf89cb94b-cpg2r"]
Nov 01 03:42:26.582176 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.582153    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tzlf4\""
Nov 01 03:42:26.582277 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.582218    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 01 03:42:26.582277 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.582266    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bv2p7\""
Nov 01 03:42:26.582388 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.582292    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-5d67x\""
Nov 01 03:42:26.582388 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.582385    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 01 03:42:26.583399 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.583378    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-twwg4"]
Nov 01 03:42:26.625790 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.625754    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-trusted-ca-bundle\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.625947 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.625805    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3766a002-acc6-4ec5-a438-c5157d248974-registry-tls\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.625947 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.625837    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sms5r\" (UniqueName: \"kubernetes.io/projected/860da2d8-6013-4648-9654-68f954815d2f-kube-api-access-sms5r\") pod \"migrator-66df5d496b-twwg4\" (UID: \"860da2d8-6013-4648-9654-68f954815d2f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twwg4"
Nov 01 03:42:26.625947 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.625863    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-oauth-config\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.625947 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.625884    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3766a002-acc6-4ec5-a438-c5157d248974-trusted-ca\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.626131 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.625986    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlnlz\" (UniqueName: \"kubernetes.io/projected/3766a002-acc6-4ec5-a438-c5157d248974-kube-api-access-tlnlz\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.626131 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626024    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stj4p\" (UniqueName: \"kubernetes.io/projected/e8730c16-4368-42d8-ba5a-78a3a7e59d6a-kube-api-access-stj4p\") pod \"migrator-66df5d496b-r7v4n\" (UID: \"e8730c16-4368-42d8-ba5a-78a3a7e59d6a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r7v4n"
Nov 01 03:42:26.626131 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626116    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-serving-cert\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.626261 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626143    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-service-ca\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.626261 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626169    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3766a002-acc6-4ec5-a438-c5157d248974-registry-certificates\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.626261 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626205    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-trusted-ca-bundle\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.626371 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626271    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3766a002-acc6-4ec5-a438-c5157d248974-bound-sa-token\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.626371 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626307    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-service-ca-bundle\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.626371 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626326    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llfg\" (UniqueName: \"kubernetes.io/projected/b96d2df2-07b5-46db-823b-5dded530efe3-kube-api-access-7llfg\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.626371 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626352    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b96d2df2-07b5-46db-823b-5dded530efe3-archives-path\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.626551 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626373    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3766a002-acc6-4ec5-a438-c5157d248974-image-registry-private-configuration\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.626551 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626392    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-oauth-serving-cert\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.626551 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626501    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3766a002-acc6-4ec5-a438-c5157d248974-ca-trust-extracted\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.626551 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626507    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-trusted-ca-bundle\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.626747 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626615    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3766a002-acc6-4ec5-a438-c5157d248974-installation-pull-secrets\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.626747 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626655    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-console-config\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.626747 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626686    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf-tmp\") pod \"downloads-6dd789c8b7-99c76\" (UID: \"b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf\") " pod="openshift-console/downloads-6dd789c8b7-99c76"
Nov 01 03:42:26.626881 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626753    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b96d2df2-07b5-46db-823b-5dded530efe3-archives-path\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.626881 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626771    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhpqz\" (UniqueName: \"kubernetes.io/projected/b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf-kube-api-access-nhpqz\") pod \"downloads-6dd789c8b7-99c76\" (UID: \"b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf\") " pod="openshift-console/downloads-6dd789c8b7-99c76"
Nov 01 03:42:26.626881 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626804    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-service-ca-bundle\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.626881 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626829    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76182662-de17-4daf-8453-a5e9b3355f5c-tmp\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.627078 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.626977    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-archives-path\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.627078 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.627010    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhkfm\" (UniqueName: \"kubernetes.io/projected/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-kube-api-access-rhkfm\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.627078 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.627023    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-trusted-ca-bundle\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.627078 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.627049    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7dlh\" (UniqueName: \"kubernetes.io/projected/76182662-de17-4daf-8453-a5e9b3355f5c-kube-api-access-k7dlh\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.627078 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.627063    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-service-ca-bundle\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.627347 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.627329    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-archives-path\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.627401 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.627332    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-service-ca-bundle\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.683736 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.683675    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhkfm\" (UniqueName: \"kubernetes.io/projected/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-kube-api-access-rhkfm\") pod \"periodic-gathering-gj465-dgxzq\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") " pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.683884 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.683863    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llfg\" (UniqueName: \"kubernetes.io/projected/b96d2df2-07b5-46db-823b-5dded530efe3-kube-api-access-7llfg\") pod \"periodic-gathering-g9bvp-mhv7w\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") " pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.715086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.715063    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:42:26.718907 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.718885    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:42:26.728115 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728087    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-serving-cert\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.728208 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728126    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-service-ca\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.728306 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728277    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3766a002-acc6-4ec5-a438-c5157d248974-registry-certificates\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.728362 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728350    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3766a002-acc6-4ec5-a438-c5157d248974-bound-sa-token\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.728415 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728395    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3766a002-acc6-4ec5-a438-c5157d248974-image-registry-private-configuration\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.728500 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728443    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-oauth-serving-cert\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.728500 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728475    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3766a002-acc6-4ec5-a438-c5157d248974-ca-trust-extracted\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.728599 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728515    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3766a002-acc6-4ec5-a438-c5157d248974-installation-pull-secrets\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.728599 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728544    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-console-config\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.728599 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728577    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf-tmp\") pod \"downloads-6dd789c8b7-99c76\" (UID: \"b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf\") " pod="openshift-console/downloads-6dd789c8b7-99c76"
Nov 01 03:42:26.728742 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728621    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhpqz\" (UniqueName: \"kubernetes.io/projected/b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf-kube-api-access-nhpqz\") pod \"downloads-6dd789c8b7-99c76\" (UID: \"b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf\") " pod="openshift-console/downloads-6dd789c8b7-99c76"
Nov 01 03:42:26.728742 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728655    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76182662-de17-4daf-8453-a5e9b3355f5c-tmp\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.728742 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728697    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dlh\" (UniqueName: \"kubernetes.io/projected/76182662-de17-4daf-8453-a5e9b3355f5c-kube-api-access-k7dlh\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.728742 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728731    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3766a002-acc6-4ec5-a438-c5157d248974-registry-tls\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.728935 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728763    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sms5r\" (UniqueName: \"kubernetes.io/projected/860da2d8-6013-4648-9654-68f954815d2f-kube-api-access-sms5r\") pod \"migrator-66df5d496b-twwg4\" (UID: \"860da2d8-6013-4648-9654-68f954815d2f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twwg4"
Nov 01 03:42:26.728935 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728783    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-oauth-config\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.728935 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728807    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3766a002-acc6-4ec5-a438-c5157d248974-trusted-ca\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.728935 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728830    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-service-ca\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.729106 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.728832    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnlz\" (UniqueName: \"kubernetes.io/projected/3766a002-acc6-4ec5-a438-c5157d248974-kube-api-access-tlnlz\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.729213 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.729111    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stj4p\" (UniqueName: \"kubernetes.io/projected/e8730c16-4368-42d8-ba5a-78a3a7e59d6a-kube-api-access-stj4p\") pod \"migrator-66df5d496b-r7v4n\" (UID: \"e8730c16-4368-42d8-ba5a-78a3a7e59d6a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r7v4n"
Nov 01 03:42:26.729213 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.729175    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3766a002-acc6-4ec5-a438-c5157d248974-registry-certificates\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.729528 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.729411    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-oauth-serving-cert\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.729861 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.729649    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76182662-de17-4daf-8453-a5e9b3355f5c-tmp\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.729861 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.729789    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf-tmp\") pod \"downloads-6dd789c8b7-99c76\" (UID: \"b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf\") " pod="openshift-console/downloads-6dd789c8b7-99c76"
Nov 01 03:42:26.730243 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.730121    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3766a002-acc6-4ec5-a438-c5157d248974-ca-trust-extracted\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.731369 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.731340    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3766a002-acc6-4ec5-a438-c5157d248974-trusted-ca\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.731593 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.731572    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-console-config\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.732804 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.732782    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3766a002-acc6-4ec5-a438-c5157d248974-registry-tls\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.733008 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.732985    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-serving-cert\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.733289 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.733267    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-oauth-config\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.733375 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.733286    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3766a002-acc6-4ec5-a438-c5157d248974-installation-pull-secrets\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.733375 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.733367    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3766a002-acc6-4ec5-a438-c5157d248974-image-registry-private-configuration\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.790883 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.790860    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stj4p\" (UniqueName: \"kubernetes.io/projected/e8730c16-4368-42d8-ba5a-78a3a7e59d6a-kube-api-access-stj4p\") pod \"migrator-66df5d496b-r7v4n\" (UID: \"e8730c16-4368-42d8-ba5a-78a3a7e59d6a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r7v4n"
Nov 01 03:42:26.791898 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.791837    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3766a002-acc6-4ec5-a438-c5157d248974-bound-sa-token\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.791898 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.791864    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnlz\" (UniqueName: \"kubernetes.io/projected/3766a002-acc6-4ec5-a438-c5157d248974-kube-api-access-tlnlz\") pod \"image-registry-5cf89cb94b-cpg2r\" (UID: \"3766a002-acc6-4ec5-a438-c5157d248974\") " pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.793251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.793215    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sms5r\" (UniqueName: \"kubernetes.io/projected/860da2d8-6013-4648-9654-68f954815d2f-kube-api-access-sms5r\") pod \"migrator-66df5d496b-twwg4\" (UID: \"860da2d8-6013-4648-9654-68f954815d2f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twwg4"
Nov 01 03:42:26.797829 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.794755    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhpqz\" (UniqueName: \"kubernetes.io/projected/b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf-kube-api-access-nhpqz\") pod \"downloads-6dd789c8b7-99c76\" (UID: \"b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf\") " pod="openshift-console/downloads-6dd789c8b7-99c76"
Nov 01 03:42:26.800220 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.800194    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7dlh\" (UniqueName: \"kubernetes.io/projected/76182662-de17-4daf-8453-a5e9b3355f5c-kube-api-access-k7dlh\") pod \"console-7fffc4755-s9qkn\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") " pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:26.830967 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.830940    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r7v4n"
Nov 01 03:42:26.853809 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.853785    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:26.860215 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:26.860186    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twwg4"
Nov 01 03:42:27.063358 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.063333    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-99c76"
Nov 01 03:42:27.066486 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.066467    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:27.094451 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.094253    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-g9bvp-mhv7w"]
Nov 01 03:42:27.096770 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.096743    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gj465-dgxzq"]
Nov 01 03:42:27.098562 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.098539    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:27.104085 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:27.104055    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96d2df2_07b5_46db_823b_5dded530efe3.slice/crio-9a20e240e17e1ed637c856264f71694d490e505c565bb6d3f8d10b251ac6a000 WatchSource:0}: Error finding container 9a20e240e17e1ed637c856264f71694d490e505c565bb6d3f8d10b251ac6a000: Status 404 returned error can't find the container with id 9a20e240e17e1ed637c856264f71694d490e505c565bb6d3f8d10b251ac6a000
Nov 01 03:42:27.106497 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:27.105540    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d33c2fc_f329_42cc_a8f5_7e0b1f2fb3ba.slice/crio-a46d67ce52ba90a8fcc7ac22208c863bdab268537ac991111c89e0a0613dddd4 WatchSource:0}: Error finding container a46d67ce52ba90a8fcc7ac22208c863bdab268537ac991111c89e0a0613dddd4: Status 404 returned error can't find the container with id a46d67ce52ba90a8fcc7ac22208c863bdab268537ac991111c89e0a0613dddd4
Nov 01 03:42:27.212123 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.212038    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r7v4n"]
Nov 01 03:42:27.221218 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:27.221184    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8730c16_4368_42d8_ba5a_78a3a7e59d6a.slice/crio-73e7145e30d0227ca1572b563a5c5de48d48d917bad3203667db7bfecaa7d4ad WatchSource:0}: Error finding container 73e7145e30d0227ca1572b563a5c5de48d48d917bad3203667db7bfecaa7d4ad: Status 404 returned error can't find the container with id 73e7145e30d0227ca1572b563a5c5de48d48d917bad3203667db7bfecaa7d4ad
Nov 01 03:42:27.248765 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.248719    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-twwg4"]
Nov 01 03:42:27.251236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.251200    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf89cb94b-cpg2r"]
Nov 01 03:42:27.256742 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:27.256696    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860da2d8_6013_4648_9654_68f954815d2f.slice/crio-85559602890cc1d0a6fedefafd9096dd6bb81916c175b9210d7fac7e3c2ecbde WatchSource:0}: Error finding container 85559602890cc1d0a6fedefafd9096dd6bb81916c175b9210d7fac7e3c2ecbde: Status 404 returned error can't find the container with id 85559602890cc1d0a6fedefafd9096dd6bb81916c175b9210d7fac7e3c2ecbde
Nov 01 03:42:27.258973 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:27.258953    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3766a002_acc6_4ec5_a438_c5157d248974.slice/crio-bb462ca8a6f28c85be837b80d719d760ccccc72294dcaed1280b5cb6d28a266a WatchSource:0}: Error finding container bb462ca8a6f28c85be837b80d719d760ccccc72294dcaed1280b5cb6d28a266a: Status 404 returned error can't find the container with id bb462ca8a6f28c85be837b80d719d760ccccc72294dcaed1280b5cb6d28a266a
Nov 01 03:42:27.444513 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.444477    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-99c76"]
Nov 01 03:42:27.447991 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.447960    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fffc4755-s9qkn"]
Nov 01 03:42:27.453026 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:27.452997    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07f6fa3_06cc_4b7c_bc7b_f5cbe1f66ebf.slice/crio-9eecc239c5f91256731eb594cd0c559b5c01431cca154c042fa25609d8aabf9d WatchSource:0}: Error finding container 9eecc239c5f91256731eb594cd0c559b5c01431cca154c042fa25609d8aabf9d: Status 404 returned error can't find the container with id 9eecc239c5f91256731eb594cd0c559b5c01431cca154c042fa25609d8aabf9d
Nov 01 03:42:27.455821 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:27.455798    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76182662_de17_4daf_8453_a5e9b3355f5c.slice/crio-d46768d5da018bc772918ccf799de7247f1dce3defb15605cfb9813036a882d5 WatchSource:0}: Error finding container d46768d5da018bc772918ccf799de7247f1dce3defb15605cfb9813036a882d5: Status 404 returned error can't find the container with id d46768d5da018bc772918ccf799de7247f1dce3defb15605cfb9813036a882d5
Nov 01 03:42:27.642851 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.642765    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:27.643075 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.642904    4882 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:42:27.644055 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.644031    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7dcc875c8f-dlqlr"
Nov 01 03:42:27.789800 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.789743    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twwg4" event={"ID":"860da2d8-6013-4648-9654-68f954815d2f","Type":"ContainerStarted","Data":"85559602890cc1d0a6fedefafd9096dd6bb81916c175b9210d7fac7e3c2ecbde"}
Nov 01 03:42:27.792080 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.792042    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r" event={"ID":"3766a002-acc6-4ec5-a438-c5157d248974","Type":"ContainerStarted","Data":"1b2a8cca918852bdb33566c6c76ed045cbb0a620efaf8bb69598a1682d93e8df"}
Nov 01 03:42:27.792589 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.792561    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r" event={"ID":"3766a002-acc6-4ec5-a438-c5157d248974","Type":"ContainerStarted","Data":"bb462ca8a6f28c85be837b80d719d760ccccc72294dcaed1280b5cb6d28a266a"}
Nov 01 03:42:27.792707 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.792620    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:27.793758 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.793733    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r7v4n" event={"ID":"e8730c16-4368-42d8-ba5a-78a3a7e59d6a","Type":"ContainerStarted","Data":"73e7145e30d0227ca1572b563a5c5de48d48d917bad3203667db7bfecaa7d4ad"}
Nov 01 03:42:27.795595 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.795563    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gj465-dgxzq" event={"ID":"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba","Type":"ContainerStarted","Data":"fd0bfd8623184bef0e2371ddabbc80425fd6a645fd4a76c617f0eb2c4766f139"}
Nov 01 03:42:27.795719 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.795703    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gj465-dgxzq" event={"ID":"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba","Type":"ContainerStarted","Data":"a46d67ce52ba90a8fcc7ac22208c863bdab268537ac991111c89e0a0613dddd4"}
Nov 01 03:42:27.797360 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.797337    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g9bvp-mhv7w" event={"ID":"b96d2df2-07b5-46db-823b-5dded530efe3","Type":"ContainerStarted","Data":"bbc934f89a477f3c98cbb4472108fcbd3514801aa793e0779fd8150bf8dfe253"}
Nov 01 03:42:27.797482 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.797379    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g9bvp-mhv7w" event={"ID":"b96d2df2-07b5-46db-823b-5dded530efe3","Type":"ContainerStarted","Data":"9a20e240e17e1ed637c856264f71694d490e505c565bb6d3f8d10b251ac6a000"}
Nov 01 03:42:27.798746 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.798726    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fffc4755-s9qkn" event={"ID":"76182662-de17-4daf-8453-a5e9b3355f5c","Type":"ContainerStarted","Data":"d46768d5da018bc772918ccf799de7247f1dce3defb15605cfb9813036a882d5"}
Nov 01 03:42:27.799880 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.799856    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-99c76" event={"ID":"b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf","Type":"ContainerStarted","Data":"9eecc239c5f91256731eb594cd0c559b5c01431cca154c042fa25609d8aabf9d"}
Nov 01 03:42:27.903015 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.902942    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx"]
Nov 01 03:42:27.907885 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.907859    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx"
Nov 01 03:42:27.908808 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.908742    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r" podStartSLOduration=1.908725695 podStartE2EDuration="1.908725695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:27.90767977 +0000 UTC m=+16.469712812" watchObservedRunningTime="2026-11-01 03:42:27.908725695 +0000 UTC m=+16.470758717"
Nov 01 03:42:27.958524 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.958495    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 03:42:27.959447 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.959406    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bkqdx\""
Nov 01 03:42:27.961104 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.960231    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx"]
Nov 01 03:42:27.965114 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:27.965091    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6d804e14-94cd-4c14-958f-66deb766ee5a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lbwxx\" (UID: \"6d804e14-94cd-4c14-958f-66deb766ee5a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx"
Nov 01 03:42:28.018919 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:28.018867    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-g9bvp-mhv7w" podStartSLOduration=2.018853283 podStartE2EDuration="2.018853283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:28.018050289 +0000 UTC m=+16.580083306" watchObservedRunningTime="2026-11-01 03:42:28.018853283 +0000 UTC m=+16.580886293"
Nov 01 03:42:28.066482 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:28.066454    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6d804e14-94cd-4c14-958f-66deb766ee5a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lbwxx\" (UID: \"6d804e14-94cd-4c14-958f-66deb766ee5a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx"
Nov 01 03:42:28.066633 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:28.066615    4882 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Nov 01 03:42:28.066711 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:28.066699    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d804e14-94cd-4c14-958f-66deb766ee5a-tls-certificates podName:6d804e14-94cd-4c14-958f-66deb766ee5a nodeName:}" failed. No retries permitted until 2026-11-01 03:42:28.566674387 +0000 UTC m=+17.128707399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/6d804e14-94cd-4c14-958f-66deb766ee5a-tls-certificates") pod "prometheus-operator-admission-webhook-6489d8bf89-lbwxx" (UID: "6d804e14-94cd-4c14-958f-66deb766ee5a") : secret "prometheus-operator-admission-webhook-tls" not found
Nov 01 03:42:28.128251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:28.128202    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-gj465-dgxzq" podStartSLOduration=2.128187991 podStartE2EDuration="2.128187991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:28.127417546 +0000 UTC m=+16.689450567" watchObservedRunningTime="2026-11-01 03:42:28.128187991 +0000 UTC m=+16.690221008"
Nov 01 03:42:28.573484 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:28.573453    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6d804e14-94cd-4c14-958f-66deb766ee5a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lbwxx\" (UID: \"6d804e14-94cd-4c14-958f-66deb766ee5a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx"
Nov 01 03:42:28.576920 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:28.576895    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6d804e14-94cd-4c14-958f-66deb766ee5a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lbwxx\" (UID: \"6d804e14-94cd-4c14-958f-66deb766ee5a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx"
Nov 01 03:42:28.821381 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:28.821306    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx"
Nov 01 03:42:29.213579 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:29.213543    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx"]
Nov 01 03:42:29.223337 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:29.223296    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d804e14_94cd_4c14_958f_66deb766ee5a.slice/crio-6921d72e54e7e52f0ef971678e9dfc879898f15788706da16ca718a55a5e51ef WatchSource:0}: Error finding container 6921d72e54e7e52f0ef971678e9dfc879898f15788706da16ca718a55a5e51ef: Status 404 returned error can't find the container with id 6921d72e54e7e52f0ef971678e9dfc879898f15788706da16ca718a55a5e51ef
Nov 01 03:42:29.795586 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:29.795513    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7v4p5_19a62e18-e602-4b06-ae13-49276b60f80c/dns/0.log"
Nov 01 03:42:29.818252 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:29.817809    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twwg4" event={"ID":"860da2d8-6013-4648-9654-68f954815d2f","Type":"ContainerStarted","Data":"efc312caf7cf90b0262cabd1d9e6842ea4450a5aa60f5486d2253d960e86598f"}
Nov 01 03:42:29.819749 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:29.819719    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r7v4n" event={"ID":"e8730c16-4368-42d8-ba5a-78a3a7e59d6a","Type":"ContainerStarted","Data":"06b519632b0c6b5c96aa91eb2053acc720233efbb84cfb8e75fdf25b4848da10"}
Nov 01 03:42:29.821263 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:29.821232    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx" event={"ID":"6d804e14-94cd-4c14-958f-66deb766ee5a","Type":"ContainerStarted","Data":"6921d72e54e7e52f0ef971678e9dfc879898f15788706da16ca718a55a5e51ef"}
Nov 01 03:42:29.828972 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:29.828954    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7v4p5_19a62e18-e602-4b06-ae13-49276b60f80c/dns/0.log"
Nov 01 03:42:29.892717 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:29.892690    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7v4p5_19a62e18-e602-4b06-ae13-49276b60f80c/kube-rbac-proxy/0.log"
Nov 01 03:42:29.926393 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:29.926337    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twwg4" podStartSLOduration=1.676187941 podStartE2EDuration="3.926317938s" totalImagesPullingTime="2.250129997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:29.925309388 +0000 UTC m=+18.487342405" watchObservedRunningTime="2026-11-01 03:42:29.926317938 +0000 UTC m=+18.488350956"
Nov 01 03:42:29.980084 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:29.980004    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7v4p5_19a62e18-e602-4b06-ae13-49276b60f80c/kube-rbac-proxy/0.log"
Nov 01 03:42:30.032976 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:30.032928    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r7v4n" podStartSLOduration=1.681796194 podStartE2EDuration="4.032913938s" totalImagesPullingTime="2.351117744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:30.031857098 +0000 UTC m=+18.593890116" watchObservedRunningTime="2026-11-01 03:42:30.032913938 +0000 UTC m=+18.594946946"
Nov 01 03:42:30.033475 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:30.033458    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t8tdn_9c80204c-9fa6-4c25-a1e6-737c7f217976/dns-node-resolver/0.log"
Nov 01 03:42:30.182150 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:30.182076    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t8tdn_9c80204c-9fa6-4c25-a1e6-737c7f217976/dns-node-resolver/0.log"
Nov 01 03:42:30.632225 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:30.632197    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-565fcbb6bc-wvnqs_c7c51022-86cd-4cac-bee9-22a33fd53e22/registry/0.log"
Nov 01 03:42:30.780628 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:30.780559    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-565fcbb6bc-wvnqs_c7c51022-86cd-4cac-bee9-22a33fd53e22/registry/0.log"
Nov 01 03:42:30.830904 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:30.830875    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cf89cb94b-cpg2r_3766a002-acc6-4ec5-a438-c5157d248974/registry/0.log"
Nov 01 03:42:30.980219 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:30.980194    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cf89cb94b-cpg2r_3766a002-acc6-4ec5-a438-c5157d248974/registry/0.log"
Nov 01 03:42:31.034509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:31.034488    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8894dd498-k59gh_6979e621-5326-4fff-8641-af89a3065735/registry/0.log"
Nov 01 03:42:31.179508 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:31.179418    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8894dd498-k59gh_6979e621-5326-4fff-8641-af89a3065735/registry/0.log"
Nov 01 03:42:31.632283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:31.632257    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fwzpl_f74d5816-8e0a-4f49-83bd-5cd907a57cab/node-ca/0.log"
Nov 01 03:42:31.780050 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:31.779978    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fwzpl_f74d5816-8e0a-4f49-83bd-5cd907a57cab/node-ca/0.log"
Nov 01 03:42:31.835500 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:31.835285    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rft8" event={"ID":"1d1984d7-7b8d-40a3-b464-ba27be829957","Type":"ContainerStarted","Data":"eb75edcfe00748190e40a0dafa29107c7a356722afcf53a2a52d02321ac9b410"}
Nov 01 03:42:31.837066 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:31.837034    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx" event={"ID":"6d804e14-94cd-4c14-958f-66deb766ee5a","Type":"ContainerStarted","Data":"4e51cb6c87446ab9e0b5b3bb49a2b220485ac35b9a17363632434861c62bdb03"}
Nov 01 03:42:31.837252 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:31.837221    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dcc875c8f-dlqlr_036c1722-9db5-406e-bc09-0aa6f215e1e0/router/0.log"
Nov 01 03:42:31.837339 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:31.837302    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx"
Nov 01 03:42:31.843012 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:31.842988    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx"
Nov 01 03:42:31.979882 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:31.979853    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dcc875c8f-dlqlr_036c1722-9db5-406e-bc09-0aa6f215e1e0/router/0.log"
Nov 01 03:42:32.031387 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.031307    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-46gkp_bd4e00b4-e891-4be2-b941-1238d43e8805/serve-healthcheck-canary/0.log"
Nov 01 03:42:32.056261 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.056210    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lbwxx" podStartSLOduration=2.808556632 podStartE2EDuration="5.056191397s" totalImagesPullingTime="2.247634765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:32.055009426 +0000 UTC m=+20.617042457" watchObservedRunningTime="2026-11-01 03:42:32.056191397 +0000 UTC m=+20.618224416"
Nov 01 03:42:32.180503 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.180471    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-46gkp_bd4e00b4-e891-4be2-b941-1238d43e8805/serve-healthcheck-canary/0.log"
Nov 01 03:42:32.232580 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.232550    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-r7v4n_e8730c16-4368-42d8-ba5a-78a3a7e59d6a/migrator/0.log"
Nov 01 03:42:32.381111 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.381033    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-r7v4n_e8730c16-4368-42d8-ba5a-78a3a7e59d6a/migrator/0.log"
Nov 01 03:42:32.432360 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.432328    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-twwg4_860da2d8-6013-4648-9654-68f954815d2f/migrator/0.log"
Nov 01 03:42:32.580616 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.580587    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-twwg4_860da2d8-6013-4648-9654-68f954815d2f/migrator/0.log"
Nov 01 03:42:32.633570 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.633485    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-w6krx_3cd84ba1-3255-444e-b2e0-2a3d4ac25988/kube-storage-version-migrator-operator/0.log"
Nov 01 03:42:32.757197 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.757164    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7v4p5"
Nov 01 03:42:32.788706 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.788655    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-w6krx_3cd84ba1-3255-444e-b2e0-2a3d4ac25988/kube-storage-version-migrator-operator/0.log"
Nov 01 03:42:32.832941 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.832911    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-w6krx_3cd84ba1-3255-444e-b2e0-2a3d4ac25988/kube-storage-version-migrator-operator/1.log"
Nov 01 03:42:32.981300 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:32.981269    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-w6krx_3cd84ba1-3255-444e-b2e0-2a3d4ac25988/kube-storage-version-migrator-operator/1.log"
Nov 01 03:42:33.850150 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:33.850109    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fffc4755-s9qkn" event={"ID":"76182662-de17-4daf-8453-a5e9b3355f5c","Type":"ContainerStarted","Data":"943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673"}
Nov 01 03:42:33.961312 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:33.961257    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7fffc4755-s9qkn" podStartSLOduration=2.4777399190000002 podStartE2EDuration="7.961240108s" totalImagesPullingTime="5.483500189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:33.960651334 +0000 UTC m=+22.522684350" watchObservedRunningTime="2026-11-01 03:42:33.961240108 +0000 UTC m=+22.523273124"
Nov 01 03:42:37.067606 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:37.067572    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:37.067992 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:37.067651    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:37.072590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:37.072568    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:37.104328 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:37.104293    4882 patch_prober.go:29] interesting pod/image-registry-8894dd498-k59gh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 03:42:37.104463 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:37.104346    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8894dd498-k59gh" podUID="6979e621-5326-4fff-8641-af89a3065735" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:42:37.875076 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:37.875048    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:42:38.244760 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:38.244724    4882 patch_prober.go:29] interesting pod/image-registry-565fcbb6bc-wvnqs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 03:42:38.245202 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:38.244784    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs" podUID="c7c51022-86cd-4cac-bee9-22a33fd53e22" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:42:39.412446 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:39.412395    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hbgqw"
Nov 01 03:42:41.512948 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:41.512917    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tws4"
Nov 01 03:42:42.277381 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.277348    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n2cmm"]
Nov 01 03:42:42.314613 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.314580    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.363203 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.363174    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 03:42:42.363408 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.363393    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 03:42:42.363491 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.363390    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 03:42:42.363491 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.363474    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ztkjl\""
Nov 01 03:42:42.363645 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.363633    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 03:42:42.403234 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.403200    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-textfile\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.403392 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.403254    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-root\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.403392 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.403296    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-metrics-client-ca\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.403392 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.403316    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.403392 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.403344    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-wtmp\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.403392 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.403374    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-tls\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.403392 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.403388    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-sys\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.403671 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.403467    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.403671 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.403487    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkb7\" (UniqueName: \"kubernetes.io/projected/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-kube-api-access-mtkb7\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.504678 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.504646    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkb7\" (UniqueName: \"kubernetes.io/projected/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-kube-api-access-mtkb7\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.504848 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.504702    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-textfile\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.504848 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.504736    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-root\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.504848 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.504780    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-metrics-client-ca\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.504848 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.504805    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.504848 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.504832    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-wtmp\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.505123 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.504854    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-root\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.505123 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.504898    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-tls\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.505123 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.504920    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-sys\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.505123 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.504977    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-wtmp\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.505123 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.504996    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.505123 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.505020    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-sys\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.505411 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.505142    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-textfile\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.505676 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.505624    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.505759 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.505677    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-metrics-client-ca\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.507689 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.507663    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-tls\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.507792 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.507729    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.562195 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.562130    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkb7\" (UniqueName: \"kubernetes.io/projected/e3a6c29e-ea5b-498a-9c23-ad13dd2464a0-kube-api-access-mtkb7\") pod \"node-exporter-n2cmm\" (UID: \"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0\") " pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.678359 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.678321    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n2cmm"
Nov 01 03:42:42.719988 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:42:42.719952    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a6c29e_ea5b_498a_9c23_ad13dd2464a0.slice/crio-3b564336024d5f7dca570e7042a7967bbec24b7cf30c876d8ef71023a8e91f36 WatchSource:0}: Error finding container 3b564336024d5f7dca570e7042a7967bbec24b7cf30c876d8ef71023a8e91f36: Status 404 returned error can't find the container with id 3b564336024d5f7dca570e7042a7967bbec24b7cf30c876d8ef71023a8e91f36
Nov 01 03:42:42.895977 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:42.895886    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2cmm" event={"ID":"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0","Type":"ContainerStarted","Data":"3b564336024d5f7dca570e7042a7967bbec24b7cf30c876d8ef71023a8e91f36"}
Nov 01 03:42:43.759844 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:43.759814    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:42:45.912713 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:45.912601    4882 generic.go:356] "Generic (PLEG): container finished" podID="e3a6c29e-ea5b-498a-9c23-ad13dd2464a0" containerID="12eed382b3840d10c58b07f27509e95cc9f1258de73784e7ba2cd3bb1797fb69" exitCode=0
Nov 01 03:42:45.912713 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:45.912660    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2cmm" event={"ID":"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0","Type":"ContainerDied","Data":"12eed382b3840d10c58b07f27509e95cc9f1258de73784e7ba2cd3bb1797fb69"}
Nov 01 03:42:46.920969 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:46.920927    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2cmm" event={"ID":"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0","Type":"ContainerStarted","Data":"da4caf90bd024880e689ea3e09c7efcf8680339851ea24dd04633f642e2ca072"}
Nov 01 03:42:46.921479 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:46.920990    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2cmm" event={"ID":"e3a6c29e-ea5b-498a-9c23-ad13dd2464a0","Type":"ContainerStarted","Data":"6ea0cd5e1511a7cc46460ac784c6f31ec66d58f0d081bb6c3edc17f938fdf322"}
Nov 01 03:42:47.035797 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:47.035741    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n2cmm" podStartSLOduration=2.325871386 podStartE2EDuration="5.035724407s" totalImagesPullingTime="2.616953037s" totalInitContainerRuntime="92.899984ms" isStatefulPod=true podCreationTimestamp="2026-11-01 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:47.03530071 +0000 UTC m=+35.597333754" watchObservedRunningTime="2026-11-01 03:42:47.035724407 +0000 UTC m=+35.597757428"
Nov 01 03:42:47.103115 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:47.103088    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:47.296332 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:47.296296    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fffc4755-s9qkn"]
Nov 01 03:42:47.928780 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:47.928702    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-99c76" event={"ID":"b07f6fa3-06cc-4b7c-bc7b-f5cbe1f66ebf","Type":"ContainerStarted","Data":"777830ffb3b514906da387a601100f04b7961bcaf9a529012583cb5c0cbfa895"}
Nov 01 03:42:47.929287 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:47.928939    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-99c76"
Nov 01 03:42:47.942873 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:47.942849    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-99c76"
Nov 01 03:42:48.041542 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:48.041477    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-99c76" podStartSLOduration=2.006804184 podStartE2EDuration="22.041458263s" totalImagesPullingTime="20.034654079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:48.039570715 +0000 UTC m=+36.601603736" watchObservedRunningTime="2026-11-01 03:42:48.041458263 +0000 UTC m=+36.603491280"
Nov 01 03:42:49.828551 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:49.828522    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cf89cb94b-cpg2r"
Nov 01 03:42:50.049815 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.049778    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-565fcbb6bc-wvnqs"]
Nov 01 03:42:50.562517 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.558206    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bb45698bf-f7v4l"]
Nov 01 03:42:50.592829 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.592794    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.618572 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.618489    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bb45698bf-f7v4l"]
Nov 01 03:42:50.689305 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.688920    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-config\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.689305 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.688952    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-service-ca\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.689305 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.688978    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-oauth-config\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.689305 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.689012    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cb116cc-1d54-4d20-a247-e0cb37c6eade-tmp\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.689305 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.689049    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-serving-cert\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.689305 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.689066    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z6q6\" (UniqueName: \"kubernetes.io/projected/0cb116cc-1d54-4d20-a247-e0cb37c6eade-kube-api-access-8z6q6\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.689305 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.689092    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-trusted-ca-bundle\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.689305 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.689119    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-oauth-serving-cert\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.689305 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.689271    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 01 03:42:50.790649 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.790610    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-config\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.790804 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.790667    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-service-ca\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.790804 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.790713    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-oauth-config\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.790804 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.790776    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cb116cc-1d54-4d20-a247-e0cb37c6eade-tmp\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.790903 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.790808    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-serving-cert\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.790903 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.790836    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z6q6\" (UniqueName: \"kubernetes.io/projected/0cb116cc-1d54-4d20-a247-e0cb37c6eade-kube-api-access-8z6q6\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.790903 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.790876    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-trusted-ca-bundle\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.790998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.790920    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-oauth-serving-cert\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.791927 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.791893    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-oauth-serving-cert\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.792836 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.792809    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-config\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.793476 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.793401    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-service-ca\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.795598 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.794635    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cb116cc-1d54-4d20-a247-e0cb37c6eade-tmp\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.796050 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.795985    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-trusted-ca-bundle\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.796722 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.796702    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-oauth-config\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.799737 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.799694    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-serving-cert\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.865123 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.861918    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z6q6\" (UniqueName: \"kubernetes.io/projected/0cb116cc-1d54-4d20-a247-e0cb37c6eade-kube-api-access-8z6q6\") pod \"console-6bb45698bf-f7v4l\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") " pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:50.930670 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:50.930636    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:42:51.380508 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:51.380380    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bb45698bf-f7v4l"]
Nov 01 03:42:51.507421 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:51.507346    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8894dd498-k59gh" podUID="6979e621-5326-4fff-8641-af89a3065735" containerName="registry" containerID="cri-o://fd4dd5d559c5421f9c558819e7e4ca68dbb33dadd9612dc1c03cb35ec86dbf7a" gracePeriod=30
Nov 01 03:42:51.982623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:51.982586    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb45698bf-f7v4l" event={"ID":"0cb116cc-1d54-4d20-a247-e0cb37c6eade","Type":"ContainerStarted","Data":"ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b"}
Nov 01 03:42:51.983112 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:51.982650    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb45698bf-f7v4l" event={"ID":"0cb116cc-1d54-4d20-a247-e0cb37c6eade","Type":"ContainerStarted","Data":"cedd0b68c2df7951bf1f1ac567e4d083c8eb8f47d5977da4ea087e514a7997e9"}
Nov 01 03:42:51.986023 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:51.985890    4882 generic.go:356] "Generic (PLEG): container finished" podID="6979e621-5326-4fff-8641-af89a3065735" containerID="fd4dd5d559c5421f9c558819e7e4ca68dbb33dadd9612dc1c03cb35ec86dbf7a" exitCode=0
Nov 01 03:42:51.986023 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:51.985952    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8894dd498-k59gh" event={"ID":"6979e621-5326-4fff-8641-af89a3065735","Type":"ContainerDied","Data":"fd4dd5d559c5421f9c558819e7e4ca68dbb33dadd9612dc1c03cb35ec86dbf7a"}
Nov 01 03:42:51.986023 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:51.985981    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8894dd498-k59gh" event={"ID":"6979e621-5326-4fff-8641-af89a3065735","Type":"ContainerDied","Data":"f786eae1a5e2176c06131cd2315a87595f105deeb22989148e3e0e82581e627b"}
Nov 01 03:42:51.986023 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:51.985995    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f786eae1a5e2176c06131cd2315a87595f105deeb22989148e3e0e82581e627b"
Nov 01 03:42:51.987727 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:51.987708    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:52.096463 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.095095    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6bb45698bf-f7v4l" podStartSLOduration=2.095072048 podStartE2EDuration="2.095072048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:52.093275136 +0000 UTC m=+40.655308154" watchObservedRunningTime="2026-11-01 03:42:52.095072048 +0000 UTC m=+40.657105067"
Nov 01 03:42:52.121652 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.121618    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-kube-api-access-kqjhm\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-kube-api-access-kqjhm\") pod \"6979e621-5326-4fff-8641-af89a3065735\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") "
Nov 01 03:42:52.121941 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.121916    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-image-registry-private-configuration\") pod \"6979e621-5326-4fff-8641-af89a3065735\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") "
Nov 01 03:42:52.122098 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.122080    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-trusted-ca\") pod \"6979e621-5326-4fff-8641-af89a3065735\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") "
Nov 01 03:42:52.122234 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.122217    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6979e621-5326-4fff-8641-af89a3065735-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6979e621-5326-4fff-8641-af89a3065735-ca-trust-extracted\") pod \"6979e621-5326-4fff-8641-af89a3065735\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") "
Nov 01 03:42:52.122357 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.122343    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-bound-sa-token\") pod \"6979e621-5326-4fff-8641-af89a3065735\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") "
Nov 01 03:42:52.122499 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.122483    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-registry-certificates\") pod \"6979e621-5326-4fff-8641-af89a3065735\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") "
Nov 01 03:42:52.122651 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.122637    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-installation-pull-secrets\") pod \"6979e621-5326-4fff-8641-af89a3065735\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") "
Nov 01 03:42:52.122776 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.122759    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") pod \"6979e621-5326-4fff-8641-af89a3065735\" (UID: \"6979e621-5326-4fff-8641-af89a3065735\") "
Nov 01 03:42:52.124045 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.124019    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-trusted-ca" pod "6979e621-5326-4fff-8641-af89a3065735" (UID: "6979e621-5326-4fff-8641-af89a3065735"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:52.128925 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.128885    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-registry-certificates" pod "6979e621-5326-4fff-8641-af89a3065735" (UID: "6979e621-5326-4fff-8641-af89a3065735"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:52.147566 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.146341    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls" pod "6979e621-5326-4fff-8641-af89a3065735" (UID: "6979e621-5326-4fff-8641-af89a3065735"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:52.148661 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.148595    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-bound-sa-token" pod "6979e621-5326-4fff-8641-af89a3065735" (UID: "6979e621-5326-4fff-8641-af89a3065735"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:52.155488 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.149344    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-installation-pull-secrets" pod "6979e621-5326-4fff-8641-af89a3065735" (UID: "6979e621-5326-4fff-8641-af89a3065735"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:42:52.155488 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.149731    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-kube-api-access-kqjhm" pod "6979e621-5326-4fff-8641-af89a3065735" (UID: "6979e621-5326-4fff-8641-af89a3065735"). InnerVolumeSpecName "kube-api-access-kqjhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:52.155488 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.152141    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6979e621-5326-4fff-8641-af89a3065735-ca-trust-extracted" pod "6979e621-5326-4fff-8641-af89a3065735" (UID: "6979e621-5326-4fff-8641-af89a3065735"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:52.155708 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.155530    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-image-registry-private-configuration" pod "6979e621-5326-4fff-8641-af89a3065735" (UID: "6979e621-5326-4fff-8641-af89a3065735"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:42:52.230449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.228013    4882 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-trusted-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:42:52.230449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.228052    4882 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6979e621-5326-4fff-8641-af89a3065735-ca-trust-extracted\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:42:52.230449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.228066    4882 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-bound-sa-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:42:52.230449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.228081    4882 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6979e621-5326-4fff-8641-af89a3065735-registry-certificates\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:42:52.230449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.228098    4882 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-installation-pull-secrets\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:42:52.230449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.228112    4882 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-registry-tls\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:42:52.230449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.228129    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqjhm\" (UniqueName: \"kubernetes.io/projected/6979e621-5326-4fff-8641-af89a3065735-kube-api-access-kqjhm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:42:52.230449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.228145    4882 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6979e621-5326-4fff-8641-af89a3065735-image-registry-private-configuration\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:42:52.992491 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:52.992456    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8894dd498-k59gh"
Nov 01 03:42:53.197155 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:53.197125    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8894dd498-k59gh"]
Nov 01 03:42:53.203452 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:53.202021    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8894dd498-k59gh"]
Nov 01 03:42:53.608078 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:53.607992    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6979e621-5326-4fff-8641-af89a3065735" path="/var/lib/kubelet/pods/6979e621-5326-4fff-8641-af89a3065735/volumes"
Nov 01 03:42:58.510557 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:42:58.510503    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bb45698bf-f7v4l"]
Nov 01 03:42:58.723680 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:42:58.723646    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:43:00.931699 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:00.931671    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:43:12.317380 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.317309    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7fffc4755-s9qkn" podUID="76182662-de17-4daf-8453-a5e9b3355f5c" containerName="console" containerID="cri-o://943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673" gracePeriod=15
Nov 01 03:43:12.568747 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.568682    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:43:12.639285 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.639253    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-oauth-config\") pod \"76182662-de17-4daf-8453-a5e9b3355f5c\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") "
Nov 01 03:43:12.639467 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.639310    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-serving-cert\") pod \"76182662-de17-4daf-8453-a5e9b3355f5c\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") "
Nov 01 03:43:12.639467 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.639339    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-console-config\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-console-config\") pod \"76182662-de17-4daf-8453-a5e9b3355f5c\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") "
Nov 01 03:43:12.639467 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.639369    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-oauth-serving-cert\") pod \"76182662-de17-4daf-8453-a5e9b3355f5c\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") "
Nov 01 03:43:12.639467 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.639439    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-service-ca\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-service-ca\") pod \"76182662-de17-4daf-8453-a5e9b3355f5c\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") "
Nov 01 03:43:12.639687 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.639482    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76182662-de17-4daf-8453-a5e9b3355f5c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/76182662-de17-4daf-8453-a5e9b3355f5c-tmp\") pod \"76182662-de17-4daf-8453-a5e9b3355f5c\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") "
Nov 01 03:43:12.639687 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.639545    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76182662-de17-4daf-8453-a5e9b3355f5c-kube-api-access-k7dlh\" (UniqueName: \"kubernetes.io/projected/76182662-de17-4daf-8453-a5e9b3355f5c-kube-api-access-k7dlh\") pod \"76182662-de17-4daf-8453-a5e9b3355f5c\" (UID: \"76182662-de17-4daf-8453-a5e9b3355f5c\") "
Nov 01 03:43:12.639878 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.639847    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-oauth-serving-cert" pod "76182662-de17-4daf-8453-a5e9b3355f5c" (UID: "76182662-de17-4daf-8453-a5e9b3355f5c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:12.639941 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.639854    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-console-config" pod "76182662-de17-4daf-8453-a5e9b3355f5c" (UID: "76182662-de17-4daf-8453-a5e9b3355f5c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:12.639941 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.639891    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-service-ca" pod "76182662-de17-4daf-8453-a5e9b3355f5c" (UID: "76182662-de17-4daf-8453-a5e9b3355f5c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:12.640133 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.640106    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76182662-de17-4daf-8453-a5e9b3355f5c-tmp" pod "76182662-de17-4daf-8453-a5e9b3355f5c" (UID: "76182662-de17-4daf-8453-a5e9b3355f5c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:12.641637 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.641609    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-oauth-config" pod "76182662-de17-4daf-8453-a5e9b3355f5c" (UID: "76182662-de17-4daf-8453-a5e9b3355f5c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:43:12.641637 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.641629    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-serving-cert" pod "76182662-de17-4daf-8453-a5e9b3355f5c" (UID: "76182662-de17-4daf-8453-a5e9b3355f5c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:43:12.641765 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.641680    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76182662-de17-4daf-8453-a5e9b3355f5c-kube-api-access-k7dlh" pod "76182662-de17-4daf-8453-a5e9b3355f5c" (UID: "76182662-de17-4daf-8453-a5e9b3355f5c"). InnerVolumeSpecName "kube-api-access-k7dlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:12.740792 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.740757    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7dlh\" (UniqueName: \"kubernetes.io/projected/76182662-de17-4daf-8453-a5e9b3355f5c-kube-api-access-k7dlh\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:12.740792 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.740786    4882 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-oauth-config\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:12.740792 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.740797    4882 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76182662-de17-4daf-8453-a5e9b3355f5c-console-serving-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:12.741012 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.740806    4882 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-console-config\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:12.741012 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.740814    4882 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-oauth-serving-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:12.741012 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.740822    4882 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76182662-de17-4daf-8453-a5e9b3355f5c-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:12.741012 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:12.740831    4882 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76182662-de17-4daf-8453-a5e9b3355f5c-tmp\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:13.075108 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:13.075072    4882 generic.go:356] "Generic (PLEG): container finished" podID="76182662-de17-4daf-8453-a5e9b3355f5c" containerID="943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673" exitCode=0
Nov 01 03:43:13.075265 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:13.075127    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fffc4755-s9qkn"
Nov 01 03:43:13.075265 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:13.075132    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fffc4755-s9qkn" event={"ID":"76182662-de17-4daf-8453-a5e9b3355f5c","Type":"ContainerDied","Data":"943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673"}
Nov 01 03:43:13.075265 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:13.075183    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fffc4755-s9qkn" event={"ID":"76182662-de17-4daf-8453-a5e9b3355f5c","Type":"ContainerDied","Data":"d46768d5da018bc772918ccf799de7247f1dce3defb15605cfb9813036a882d5"}
Nov 01 03:43:13.075265 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:13.075202    4882 scope.go:122] "RemoveContainer" containerID="943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673"
Nov 01 03:43:13.087680 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:13.087662    4882 scope.go:122] "RemoveContainer" containerID="943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673"
Nov 01 03:43:13.087960 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:43:13.087939    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673\": container with ID starting with 943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673 not found: ID does not exist" containerID="943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673"
Nov 01 03:43:13.088008 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:13.087969    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673"} err="failed to get container status \"943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673\": rpc error: code = NotFound desc = could not find container \"943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673\": container with ID starting with 943b118d4ad34faa8e15736b16f061ae49255247d2b793ee44529b2749ea6673 not found: ID does not exist"
Nov 01 03:43:13.230137 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:13.230108    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fffc4755-s9qkn"]
Nov 01 03:43:13.234283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:13.234255    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fffc4755-s9qkn"]
Nov 01 03:43:13.597996 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:13.597966    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76182662-de17-4daf-8453-a5e9b3355f5c" path="/var/lib/kubelet/pods/76182662-de17-4daf-8453-a5e9b3355f5c/volumes"
Nov 01 03:43:15.090389 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.090330    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs" podUID="c7c51022-86cd-4cac-bee9-22a33fd53e22" containerName="registry" containerID="cri-o://bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a" gracePeriod=30
Nov 01 03:43:15.349615 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.349560    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:43:15.364897 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.364876    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-trusted-ca\") pod \"c7c51022-86cd-4cac-bee9-22a33fd53e22\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") "
Nov 01 03:43:15.364956 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.364916    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-image-registry-private-configuration\") pod \"c7c51022-86cd-4cac-bee9-22a33fd53e22\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") "
Nov 01 03:43:15.364996 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.364966    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c51022-86cd-4cac-bee9-22a33fd53e22-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7c51022-86cd-4cac-bee9-22a33fd53e22-ca-trust-extracted\") pod \"c7c51022-86cd-4cac-bee9-22a33fd53e22\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") "
Nov 01 03:43:15.365033 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.364996    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-bound-sa-token\") pod \"c7c51022-86cd-4cac-bee9-22a33fd53e22\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") "
Nov 01 03:43:15.365085 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.365032    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-installation-pull-secrets\") pod \"c7c51022-86cd-4cac-bee9-22a33fd53e22\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") "
Nov 01 03:43:15.365143 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.365090    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-certificates\") pod \"c7c51022-86cd-4cac-bee9-22a33fd53e22\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") "
Nov 01 03:43:15.365198 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.365144    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") pod \"c7c51022-86cd-4cac-bee9-22a33fd53e22\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") "
Nov 01 03:43:15.365198 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.365178    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-kube-api-access-msg64\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-kube-api-access-msg64\") pod \"c7c51022-86cd-4cac-bee9-22a33fd53e22\" (UID: \"c7c51022-86cd-4cac-bee9-22a33fd53e22\") "
Nov 01 03:43:15.365601 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.365328    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-trusted-ca" pod "c7c51022-86cd-4cac-bee9-22a33fd53e22" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:15.365601 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.365585    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-certificates" pod "c7c51022-86cd-4cac-bee9-22a33fd53e22" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:15.367288 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.367258    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-image-registry-private-configuration" pod "c7c51022-86cd-4cac-bee9-22a33fd53e22" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:43:15.367502 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.367482    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-kube-api-access-msg64" pod "c7c51022-86cd-4cac-bee9-22a33fd53e22" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22"). InnerVolumeSpecName "kube-api-access-msg64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:15.367715 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.367684    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-installation-pull-secrets" pod "c7c51022-86cd-4cac-bee9-22a33fd53e22" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:43:15.367819 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.367793    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-bound-sa-token" pod "c7c51022-86cd-4cac-bee9-22a33fd53e22" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:15.367985 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.367966    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls" pod "c7c51022-86cd-4cac-bee9-22a33fd53e22" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:15.372109 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.372090    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c51022-86cd-4cac-bee9-22a33fd53e22-ca-trust-extracted" pod "c7c51022-86cd-4cac-bee9-22a33fd53e22" (UID: "c7c51022-86cd-4cac-bee9-22a33fd53e22"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:15.466505 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.466479    4882 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-bound-sa-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:15.466505 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.466503    4882 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-installation-pull-secrets\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:15.466660 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.466513    4882 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-certificates\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:15.466660 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.466524    4882 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-registry-tls\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:15.466660 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.466533    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msg64\" (UniqueName: \"kubernetes.io/projected/c7c51022-86cd-4cac-bee9-22a33fd53e22-kube-api-access-msg64\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:15.466660 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.466541    4882 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7c51022-86cd-4cac-bee9-22a33fd53e22-trusted-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:15.466660 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.466550    4882 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7c51022-86cd-4cac-bee9-22a33fd53e22-image-registry-private-configuration\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:15.466660 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:15.466559    4882 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7c51022-86cd-4cac-bee9-22a33fd53e22-ca-trust-extracted\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:16.091549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:16.091514    4882 generic.go:356] "Generic (PLEG): container finished" podID="c7c51022-86cd-4cac-bee9-22a33fd53e22" containerID="bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a" exitCode=0
Nov 01 03:43:16.091967 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:16.091559    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs"
Nov 01 03:43:16.091967 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:16.091570    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs" event={"ID":"c7c51022-86cd-4cac-bee9-22a33fd53e22","Type":"ContainerDied","Data":"bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a"}
Nov 01 03:43:16.091967 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:16.091608    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565fcbb6bc-wvnqs" event={"ID":"c7c51022-86cd-4cac-bee9-22a33fd53e22","Type":"ContainerDied","Data":"2a58f9d22e46b035d548ea7622a7cba7a9d294128cde5f23e40f170e3f454999"}
Nov 01 03:43:16.091967 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:16.091627    4882 scope.go:122] "RemoveContainer" containerID="bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a"
Nov 01 03:43:16.099972 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:16.099937    4882 scope.go:122] "RemoveContainer" containerID="bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a"
Nov 01 03:43:16.100259 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:43:16.100242    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a\": container with ID starting with bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a not found: ID does not exist" containerID="bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a"
Nov 01 03:43:16.100307 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:16.100269    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a"} err="failed to get container status \"bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a\": rpc error: code = NotFound desc = could not find container \"bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a\": container with ID starting with bde3fdb7b4c3ec942a499c7090532b393cba87be575cb1f01d0fe91a2b35b14a not found: ID does not exist"
Nov 01 03:43:16.305847 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:16.305814    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-565fcbb6bc-wvnqs"]
Nov 01 03:43:16.352114 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:16.352036    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-565fcbb6bc-wvnqs"]
Nov 01 03:43:17.602992 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:17.602963    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c51022-86cd-4cac-bee9-22a33fd53e22" path="/var/lib/kubelet/pods/c7c51022-86cd-4cac-bee9-22a33fd53e22/volumes"
Nov 01 03:43:23.530377 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.530334    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6bb45698bf-f7v4l" podUID="0cb116cc-1d54-4d20-a247-e0cb37c6eade" containerName="console" containerID="cri-o://ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b" gracePeriod=15
Nov 01 03:43:23.834742 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.834718    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:43:23.941116 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941089    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-service-ca\") pod \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") "
Nov 01 03:43:23.941272 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941135    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-trusted-ca-bundle\") pod \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") "
Nov 01 03:43:23.941272 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941179    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb116cc-1d54-4d20-a247-e0cb37c6eade-kube-api-access-8z6q6\" (UniqueName: \"kubernetes.io/projected/0cb116cc-1d54-4d20-a247-e0cb37c6eade-kube-api-access-8z6q6\") pod \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") "
Nov 01 03:43:23.941272 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941208    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-serving-cert\") pod \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") "
Nov 01 03:43:23.941272 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941237    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-config\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-config\") pod \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") "
Nov 01 03:43:23.941516 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941299    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb116cc-1d54-4d20-a247-e0cb37c6eade-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cb116cc-1d54-4d20-a247-e0cb37c6eade-tmp\") pod \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") "
Nov 01 03:43:23.941516 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941336    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-oauth-serving-cert\") pod \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") "
Nov 01 03:43:23.941516 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941368    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-oauth-config\") pod \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\" (UID: \"0cb116cc-1d54-4d20-a247-e0cb37c6eade\") "
Nov 01 03:43:23.941670 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941609    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-service-ca" pod "0cb116cc-1d54-4d20-a247-e0cb37c6eade" (UID: "0cb116cc-1d54-4d20-a247-e0cb37c6eade"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:23.941670 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941629    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-trusted-ca-bundle" pod "0cb116cc-1d54-4d20-a247-e0cb37c6eade" (UID: "0cb116cc-1d54-4d20-a247-e0cb37c6eade"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:23.941780 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941723    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb116cc-1d54-4d20-a247-e0cb37c6eade-tmp" pod "0cb116cc-1d54-4d20-a247-e0cb37c6eade" (UID: "0cb116cc-1d54-4d20-a247-e0cb37c6eade"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:23.941780 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941746    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-config" pod "0cb116cc-1d54-4d20-a247-e0cb37c6eade" (UID: "0cb116cc-1d54-4d20-a247-e0cb37c6eade"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:23.941966 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.941945    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-oauth-serving-cert" pod "0cb116cc-1d54-4d20-a247-e0cb37c6eade" (UID: "0cb116cc-1d54-4d20-a247-e0cb37c6eade"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:23.943465 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.943416    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-serving-cert" pod "0cb116cc-1d54-4d20-a247-e0cb37c6eade" (UID: "0cb116cc-1d54-4d20-a247-e0cb37c6eade"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:43:23.943573 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.943489    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb116cc-1d54-4d20-a247-e0cb37c6eade-kube-api-access-8z6q6" pod "0cb116cc-1d54-4d20-a247-e0cb37c6eade" (UID: "0cb116cc-1d54-4d20-a247-e0cb37c6eade"). InnerVolumeSpecName "kube-api-access-8z6q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:23.943573 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:23.943507    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-oauth-config" pod "0cb116cc-1d54-4d20-a247-e0cb37c6eade" (UID: "0cb116cc-1d54-4d20-a247-e0cb37c6eade"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:43:24.042616 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.042589    4882 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cb116cc-1d54-4d20-a247-e0cb37c6eade-tmp\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:24.042616 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.042612    4882 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-oauth-serving-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:24.042616 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.042621    4882 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-oauth-config\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:24.042812 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.042632    4882 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:24.042812 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.042641    4882 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-trusted-ca-bundle\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:24.042812 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.042649    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z6q6\" (UniqueName: \"kubernetes.io/projected/0cb116cc-1d54-4d20-a247-e0cb37c6eade-kube-api-access-8z6q6\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:24.042812 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.042658    4882 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-serving-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:24.042812 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.042667    4882 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cb116cc-1d54-4d20-a247-e0cb37c6eade-console-config\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:24.127310 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.127236    4882 generic.go:356] "Generic (PLEG): container finished" podID="0cb116cc-1d54-4d20-a247-e0cb37c6eade" containerID="ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b" exitCode=0
Nov 01 03:43:24.127310 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.127285    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb45698bf-f7v4l"
Nov 01 03:43:24.127494 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.127324    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb45698bf-f7v4l" event={"ID":"0cb116cc-1d54-4d20-a247-e0cb37c6eade","Type":"ContainerDied","Data":"ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b"}
Nov 01 03:43:24.127494 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.127378    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb45698bf-f7v4l" event={"ID":"0cb116cc-1d54-4d20-a247-e0cb37c6eade","Type":"ContainerDied","Data":"cedd0b68c2df7951bf1f1ac567e4d083c8eb8f47d5977da4ea087e514a7997e9"}
Nov 01 03:43:24.127494 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.127401    4882 scope.go:122] "RemoveContainer" containerID="ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b"
Nov 01 03:43:24.136644 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.136624    4882 scope.go:122] "RemoveContainer" containerID="ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b"
Nov 01 03:43:24.136915 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:43:24.136893    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b\": container with ID starting with ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b not found: ID does not exist" containerID="ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b"
Nov 01 03:43:24.137003 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.136921    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b"} err="failed to get container status \"ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b\": rpc error: code = NotFound desc = could not find container \"ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b\": container with ID starting with ca3387c59a66b6b6ac946338c05ea2c91e7ee9e258dbad32527d41004960f48b not found: ID does not exist"
Nov 01 03:43:24.280098 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.280065    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bb45698bf-f7v4l"]
Nov 01 03:43:24.288770 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:24.288747    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bb45698bf-f7v4l"]
Nov 01 03:43:25.597365 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:25.597330    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb116cc-1d54-4d20-a247-e0cb37c6eade" path="/var/lib/kubelet/pods/0cb116cc-1d54-4d20-a247-e0cb37c6eade/volumes"
Nov 01 03:43:28.721718 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:43:28.721678    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:43:57.270771 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:57.270732    4882 generic.go:356] "Generic (PLEG): container finished" podID="0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba" containerID="fd0bfd8623184bef0e2371ddabbc80425fd6a645fd4a76c617f0eb2c4766f139" exitCode=0
Nov 01 03:43:57.271251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:57.270803    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gj465-dgxzq" event={"ID":"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba","Type":"ContainerDied","Data":"fd0bfd8623184bef0e2371ddabbc80425fd6a645fd4a76c617f0eb2c4766f139"}
Nov 01 03:43:57.272166 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:57.272148    4882 generic.go:356] "Generic (PLEG): container finished" podID="b96d2df2-07b5-46db-823b-5dded530efe3" containerID="bbc934f89a477f3c98cbb4472108fcbd3514801aa793e0779fd8150bf8dfe253" exitCode=0
Nov 01 03:43:57.272266 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:57.272223    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g9bvp-mhv7w" event={"ID":"b96d2df2-07b5-46db-823b-5dded530efe3","Type":"ContainerDied","Data":"bbc934f89a477f3c98cbb4472108fcbd3514801aa793e0779fd8150bf8dfe253"}
Nov 01 03:43:58.467680 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.467649    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:43:58.469630 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.469613    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:43:58.533545 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.533466    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96d2df2-07b5-46db-823b-5dded530efe3-kube-api-access-7llfg\" (UniqueName: \"kubernetes.io/projected/b96d2df2-07b5-46db-823b-5dded530efe3-kube-api-access-7llfg\") pod \"b96d2df2-07b5-46db-823b-5dded530efe3\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") "
Nov 01 03:43:58.533545 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.533510    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-service-ca-bundle\") pod \"b96d2df2-07b5-46db-823b-5dded530efe3\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") "
Nov 01 03:43:58.533741 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.533555    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-service-ca-bundle\") pod \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") "
Nov 01 03:43:58.533741 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.533574    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-archives-path\") pod \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") "
Nov 01 03:43:58.533741 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.533593    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b96d2df2-07b5-46db-823b-5dded530efe3-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b96d2df2-07b5-46db-823b-5dded530efe3-archives-path\") pod \"b96d2df2-07b5-46db-823b-5dded530efe3\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") "
Nov 01 03:43:58.533741 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.533615    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-trusted-ca-bundle\") pod \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") "
Nov 01 03:43:58.533741 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.533646    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-kube-api-access-rhkfm\" (UniqueName: \"kubernetes.io/projected/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-kube-api-access-rhkfm\") pod \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\" (UID: \"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba\") "
Nov 01 03:43:58.533741 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.533671    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-trusted-ca-bundle\") pod \"b96d2df2-07b5-46db-823b-5dded530efe3\" (UID: \"b96d2df2-07b5-46db-823b-5dded530efe3\") "
Nov 01 03:43:58.534062 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.534012    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-service-ca-bundle" pod "0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba" (UID: "0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:58.534062 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.534033    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-service-ca-bundle" pod "b96d2df2-07b5-46db-823b-5dded530efe3" (UID: "b96d2df2-07b5-46db-823b-5dded530efe3"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:58.534062 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.534013    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-archives-path" pod "0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba" (UID: "0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:58.534212 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.534176    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-trusted-ca-bundle" pod "0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba" (UID: "0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:58.534260 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.534221    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96d2df2-07b5-46db-823b-5dded530efe3-archives-path" pod "b96d2df2-07b5-46db-823b-5dded530efe3" (UID: "b96d2df2-07b5-46db-823b-5dded530efe3"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:58.534260 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.534245    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-trusted-ca-bundle" pod "b96d2df2-07b5-46db-823b-5dded530efe3" (UID: "b96d2df2-07b5-46db-823b-5dded530efe3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:58.535814 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.535793    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96d2df2-07b5-46db-823b-5dded530efe3-kube-api-access-7llfg" pod "b96d2df2-07b5-46db-823b-5dded530efe3" (UID: "b96d2df2-07b5-46db-823b-5dded530efe3"). InnerVolumeSpecName "kube-api-access-7llfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:58.535870 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.535846    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-kube-api-access-rhkfm" pod "0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba" (UID: "0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba"). InnerVolumeSpecName "kube-api-access-rhkfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:58.634629 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.634595    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7llfg\" (UniqueName: \"kubernetes.io/projected/b96d2df2-07b5-46db-823b-5dded530efe3-kube-api-access-7llfg\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:58.634629 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.634623    4882 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-service-ca-bundle\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:58.634629 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.634632    4882 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-service-ca-bundle\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:58.634876 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.634642    4882 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-archives-path\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:58.634876 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.634651    4882 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b96d2df2-07b5-46db-823b-5dded530efe3-archives-path\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:58.634876 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.634659    4882 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-trusted-ca-bundle\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:58.634876 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.634667    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhkfm\" (UniqueName: \"kubernetes.io/projected/0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba-kube-api-access-rhkfm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:58.634876 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:58.634676    4882 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d2df2-07b5-46db-823b-5dded530efe3-trusted-ca-bundle\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:43:58.722863 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:43:58.722825    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:43:59.282331 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:59.282301    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gj465-dgxzq"
Nov 01 03:43:59.282564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:59.282298    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gj465-dgxzq" event={"ID":"0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba","Type":"ContainerDied","Data":"a46d67ce52ba90a8fcc7ac22208c863bdab268537ac991111c89e0a0613dddd4"}
Nov 01 03:43:59.282564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:59.282417    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a46d67ce52ba90a8fcc7ac22208c863bdab268537ac991111c89e0a0613dddd4"
Nov 01 03:43:59.283553 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:59.283517    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g9bvp-mhv7w"
Nov 01 03:43:59.283644 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:59.283519    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g9bvp-mhv7w" event={"ID":"b96d2df2-07b5-46db-823b-5dded530efe3","Type":"ContainerDied","Data":"9a20e240e17e1ed637c856264f71694d490e505c565bb6d3f8d10b251ac6a000"}
Nov 01 03:43:59.283698 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:43:59.283655    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a20e240e17e1ed637c856264f71694d490e505c565bb6d3f8d10b251ac6a000"
Nov 01 03:44:28.721114 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:44:28.721025    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:44:58.722296 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:44:58.722258    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:45:28.722005 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:45:28.721965    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:45:49.332387 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332351    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"]
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332859    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0cb116cc-1d54-4d20-a247-e0cb37c6eade" containerName="console"
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332878    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb116cc-1d54-4d20-a247-e0cb37c6eade" containerName="console"
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332911    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba" containerName="insights-gathering"
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332919    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba" containerName="insights-gathering"
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332937    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b96d2df2-07b5-46db-823b-5dded530efe3" containerName="insights-gathering"
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332944    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96d2df2-07b5-46db-823b-5dded530efe3" containerName="insights-gathering"
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332955    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6979e621-5326-4fff-8641-af89a3065735" containerName="registry"
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332963    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6979e621-5326-4fff-8641-af89a3065735" containerName="registry"
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332978    4882 cpu_manager.go:414] "containerMap: removing container" podUID="76182662-de17-4daf-8453-a5e9b3355f5c" containerName="console"
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332985    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76182662-de17-4daf-8453-a5e9b3355f5c" containerName="console"
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.332997    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c7c51022-86cd-4cac-bee9-22a33fd53e22" containerName="registry"
Nov 01 03:45:49.333032 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.333004    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c51022-86cd-4cac-bee9-22a33fd53e22" containerName="registry"
Nov 01 03:45:49.333634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.333080    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="76182662-de17-4daf-8453-a5e9b3355f5c" containerName="console"
Nov 01 03:45:49.333634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.333093    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c51022-86cd-4cac-bee9-22a33fd53e22" containerName="registry"
Nov 01 03:45:49.333634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.333107    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb116cc-1d54-4d20-a247-e0cb37c6eade" containerName="console"
Nov 01 03:45:49.333634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.333117    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d33c2fc-f329-42cc-a8f5-7e0b1f2fb3ba" containerName="insights-gathering"
Nov 01 03:45:49.333634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.333127    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96d2df2-07b5-46db-823b-5dded530efe3" containerName="insights-gathering"
Nov 01 03:45:49.333634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.333136    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6979e621-5326-4fff-8641-af89a3065735" containerName="registry"
Nov 01 03:45:49.336384 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.336363    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.384791 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.384766    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 03:45:49.389170 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.389142    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.389296 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.389187    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8vx\" (UniqueName: \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-kube-api-access-ch8vx\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.389296 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.389215    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.389296 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.389260    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.389418 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.389322    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.389418 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.389343    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.389418 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.389359    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.389418 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.389388    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732dba6f-5c7f-48bf-8802-6742d16cc51d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.389418 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.389414    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.409251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.409225    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qr8p5\""
Nov 01 03:45:49.433464 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.433412    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"]
Nov 01 03:45:49.490484 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.490454    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.490644 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.490527    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.490644 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.490558    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8vx\" (UniqueName: \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-kube-api-access-ch8vx\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.490644 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.490588    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.490644 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.490610    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.490644 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.490630    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.490889 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.490652    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.490889 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.490679    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.490889 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.490716    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732dba6f-5c7f-48bf-8802-6742d16cc51d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.491028 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.491012    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.491083 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.491045    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.491083 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.491068    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.491182 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.491130    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.491447 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.491397    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732dba6f-5c7f-48bf-8802-6742d16cc51d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.493007 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.492984    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.493449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.493413    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.550574 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.550550    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.550783 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.550760    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8vx\" (UniqueName: \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-kube-api-access-ch8vx\") pod \"maas-default-gateway-openshift-default-5df48fcc99-vjrbz\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.649329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.649246    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:49.652422 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.652395    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"]
Nov 01 03:45:49.655865 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.655846    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.704309 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.704273    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-94j4x\""
Nov 01 03:45:49.714569 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.714540    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"]
Nov 01 03:45:49.793695 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.793666    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.793856 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.793701    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.793856 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.793728    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.793856 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.793807    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.793972 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.793936    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.793972 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.793962    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.794052 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.794038    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.794093 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.794062    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.794129 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.794088    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-799tm\" (UniqueName: \"kubernetes.io/projected/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-kube-api-access-799tm\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895027 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.894996    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895185 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895035    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895185 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895057    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-799tm\" (UniqueName: \"kubernetes.io/projected/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-kube-api-access-799tm\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895185 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895108    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895185 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895141    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895185 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895170    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895478 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895211    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895478 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895249    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895478 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895273    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895644 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895495    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895697 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895642    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895844 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895817    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895990 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895877    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.895990 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.895958    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.897530 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.897512    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.897792 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.897774    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.952621 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.952599    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.952752 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.952735    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-799tm\" (UniqueName: \"kubernetes.io/projected/8d06d61e-c4e6-465b-b8f1-8de9ab96a46b-kube-api-access-799tm\") pod \"openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh\" (UID: \"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:49.977593 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:49.977563    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:50.076640 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.076608    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"]
Nov 01 03:45:50.084963 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:45:50.084933    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732dba6f_5c7f_48bf_8802_6742d16cc51d.slice/crio-832bd65b5eb3ba83318c7d0d63921fa2424b166d6a6f8902f7888bbb0f5cfb21 WatchSource:0}: Error finding container 832bd65b5eb3ba83318c7d0d63921fa2424b166d6a6f8902f7888bbb0f5cfb21: Status 404 returned error can't find the container with id 832bd65b5eb3ba83318c7d0d63921fa2424b166d6a6f8902f7888bbb0f5cfb21
Nov 01 03:45:50.350889 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.350781    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"]
Nov 01 03:45:50.359287 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:45:50.359265    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d06d61e_c4e6_465b_b8f1_8de9ab96a46b.slice/crio-e9f92282f919872fc0087854678a0b3d6017d168c155e006be3bd09d866d36bd WatchSource:0}: Error finding container e9f92282f919872fc0087854678a0b3d6017d168c155e006be3bd09d866d36bd: Status 404 returned error can't find the container with id e9f92282f919872fc0087854678a0b3d6017d168c155e006be3bd09d866d36bd
Nov 01 03:45:50.473475 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.473440    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"]
Nov 01 03:45:50.477219 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.477196    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:45:50.525950 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.525923    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 03:45:50.526081 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.525952    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 03:45:50.526081 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.525961    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-qzpzb\""
Nov 01 03:45:50.531313 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.531289    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"]
Nov 01 03:45:50.602634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.602562    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvhg\" (UniqueName: \"kubernetes.io/projected/ec6bcc76-af29-4412-9477-2399fd51b1fb-kube-api-access-gcvhg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:45:50.602634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.602598    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:45:50.602850 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.602738    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:45:50.703294 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.703258    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvhg\" (UniqueName: \"kubernetes.io/projected/ec6bcc76-af29-4412-9477-2399fd51b1fb-kube-api-access-gcvhg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:45:50.703294 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.703297    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:45:50.703565 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.703331    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:45:50.703682 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.703664    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:45:50.703728 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.703712    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:45:50.744305 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.744263    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" event={"ID":"732dba6f-5c7f-48bf-8802-6742d16cc51d","Type":"ContainerStarted","Data":"832bd65b5eb3ba83318c7d0d63921fa2424b166d6a6f8902f7888bbb0f5cfb21"}
Nov 01 03:45:50.745470 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.745419    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh" event={"ID":"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b","Type":"ContainerStarted","Data":"e9f92282f919872fc0087854678a0b3d6017d168c155e006be3bd09d866d36bd"}
Nov 01 03:45:50.760170 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.760150    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvhg\" (UniqueName: \"kubernetes.io/projected/ec6bcc76-af29-4412-9477-2399fd51b1fb-kube-api-access-gcvhg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:45:50.792613 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:50.792585    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:45:51.162690 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:51.162651    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"]
Nov 01 03:45:51.171708 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:45:51.171686    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6bcc76_af29_4412_9477_2399fd51b1fb.slice/crio-f55e45a98ab1d33e6a56beeff29441bd8c47a129e44bd68bcf7a152ac4de762e WatchSource:0}: Error finding container f55e45a98ab1d33e6a56beeff29441bd8c47a129e44bd68bcf7a152ac4de762e: Status 404 returned error can't find the container with id f55e45a98ab1d33e6a56beeff29441bd8c47a129e44bd68bcf7a152ac4de762e
Nov 01 03:45:51.750319 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:51.750283    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc" event={"ID":"ec6bcc76-af29-4412-9477-2399fd51b1fb","Type":"ContainerStarted","Data":"f55e45a98ab1d33e6a56beeff29441bd8c47a129e44bd68bcf7a152ac4de762e"}
Nov 01 03:45:56.783133 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:56.783104    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:45:56.783406 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:56.783175    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:45:56.783406 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:56.783207    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:45:57.778157 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:57.778124    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh" event={"ID":"8d06d61e-c4e6-465b-b8f1-8de9ab96a46b","Type":"ContainerStarted","Data":"091fbdb8e292203c8be8ed76ccfca298b0dc23e2f5125dff545fd73972a9abbe"}
Nov 01 03:45:57.809408 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:57.809379    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:45:57.809678 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:57.809462    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:45:57.809678 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:57.809496    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:45:57.973467 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:57.973389    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh" podStartSLOduration=2.5521852430000003 podStartE2EDuration="8.973371388s" totalImagesPullingTime="6.421186145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:57.971968658 +0000 UTC m=+226.534001674" watchObservedRunningTime="2026-11-01 03:45:57.973371388 +0000 UTC m=+226.535404409"
Nov 01 03:45:57.978486 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:57.978463    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:57.982689 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:57.982665    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:58.721856 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:45:58.721810    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:45:58.783473 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:58.783414    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" event={"ID":"732dba6f-5c7f-48bf-8802-6742d16cc51d","Type":"ContainerStarted","Data":"000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5"}
Nov 01 03:45:58.783735 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:58.783707    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:58.784541 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:58.784523    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh"
Nov 01 03:45:58.897306 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:58.897219    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" podStartSLOduration=2.175273612 podStartE2EDuration="9.897203028s" totalImagesPullingTime="7.721929416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:58.896613367 +0000 UTC m=+227.458646387" watchObservedRunningTime="2026-11-01 03:45:58.897203028 +0000 UTC m=+227.459236050"
Nov 01 03:45:59.649801 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:59.649766    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:59.654209 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:59.654186    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:59.787308 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:59.787275    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:45:59.787964 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:45:59.787945    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:46:03.806729 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:03.806691    4882 generic.go:356] "Generic (PLEG): container finished" podID="ec6bcc76-af29-4412-9477-2399fd51b1fb" containerID="99d5f25b4f62bcae03b8013ad02ff66af9dffc9dd34e4af30550f3f2794beaad" exitCode=0
Nov 01 03:46:03.807101 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:03.806770    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc" event={"ID":"ec6bcc76-af29-4412-9477-2399fd51b1fb","Type":"ContainerDied","Data":"99d5f25b4f62bcae03b8013ad02ff66af9dffc9dd34e4af30550f3f2794beaad"}
Nov 01 03:46:06.820618 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:06.820582    4882 generic.go:356] "Generic (PLEG): container finished" podID="ec6bcc76-af29-4412-9477-2399fd51b1fb" containerID="5842c77d76127b25206659a992f0c6cd6952ef0c21d36e8df5b27add4328d9af" exitCode=0
Nov 01 03:46:06.821008 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:06.820639    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc" event={"ID":"ec6bcc76-af29-4412-9477-2399fd51b1fb","Type":"ContainerDied","Data":"5842c77d76127b25206659a992f0c6cd6952ef0c21d36e8df5b27add4328d9af"}
Nov 01 03:46:20.886031 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:20.885947    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc" event={"ID":"ec6bcc76-af29-4412-9477-2399fd51b1fb","Type":"ContainerStarted","Data":"770694c3ec218afa26892cdbf787882407d62ebc216fcc9cbce40ebe38a0d96f"}
Nov 01 03:46:20.997816 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:20.997766    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc" podStartSLOduration=2.618706891 podStartE2EDuration="30.997752528s" totalImagesPullingTime="28.379045637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:46:20.996497443 +0000 UTC m=+249.558530457" watchObservedRunningTime="2026-11-01 03:46:20.997752528 +0000 UTC m=+249.559785545"
Nov 01 03:46:21.892973 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:21.892936    4882 generic.go:356] "Generic (PLEG): container finished" podID="ec6bcc76-af29-4412-9477-2399fd51b1fb" containerID="770694c3ec218afa26892cdbf787882407d62ebc216fcc9cbce40ebe38a0d96f" exitCode=0
Nov 01 03:46:21.893404 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:21.892986    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc" event={"ID":"ec6bcc76-af29-4412-9477-2399fd51b1fb","Type":"ContainerDied","Data":"770694c3ec218afa26892cdbf787882407d62ebc216fcc9cbce40ebe38a0d96f"}
Nov 01 03:46:23.026810 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.026785    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:46:23.066028 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.066004    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6bcc76-af29-4412-9477-2399fd51b1fb-kube-api-access-gcvhg\" (UniqueName: \"kubernetes.io/projected/ec6bcc76-af29-4412-9477-2399fd51b1fb-kube-api-access-gcvhg\") pod \"ec6bcc76-af29-4412-9477-2399fd51b1fb\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") "
Nov 01 03:46:23.066120 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.066048    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-bundle\") pod \"ec6bcc76-af29-4412-9477-2399fd51b1fb\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") "
Nov 01 03:46:23.066120 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.066071    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-util\" (UniqueName: \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-util\") pod \"ec6bcc76-af29-4412-9477-2399fd51b1fb\" (UID: \"ec6bcc76-af29-4412-9477-2399fd51b1fb\") "
Nov 01 03:46:23.066501 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.066473    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-bundle" pod "ec6bcc76-af29-4412-9477-2399fd51b1fb" (UID: "ec6bcc76-af29-4412-9477-2399fd51b1fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:23.068072 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.068042    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6bcc76-af29-4412-9477-2399fd51b1fb-kube-api-access-gcvhg" pod "ec6bcc76-af29-4412-9477-2399fd51b1fb" (UID: "ec6bcc76-af29-4412-9477-2399fd51b1fb"). InnerVolumeSpecName "kube-api-access-gcvhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:46:23.070286 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.070267    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-util" pod "ec6bcc76-af29-4412-9477-2399fd51b1fb" (UID: "ec6bcc76-af29-4412-9477-2399fd51b1fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:23.166835 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.166769    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcvhg\" (UniqueName: \"kubernetes.io/projected/ec6bcc76-af29-4412-9477-2399fd51b1fb-kube-api-access-gcvhg\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:46:23.166835 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.166802    4882 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-bundle\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:46:23.166835 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.166812    4882 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec6bcc76-af29-4412-9477-2399fd51b1fb-util\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:46:23.903787 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.903702    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc"
Nov 01 03:46:23.903787 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.903716    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xzrnc" event={"ID":"ec6bcc76-af29-4412-9477-2399fd51b1fb","Type":"ContainerDied","Data":"f55e45a98ab1d33e6a56beeff29441bd8c47a129e44bd68bcf7a152ac4de762e"}
Nov 01 03:46:23.903787 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:23.903769    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f55e45a98ab1d33e6a56beeff29441bd8c47a129e44bd68bcf7a152ac4de762e"
Nov 01 03:46:28.722180 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:46:28.722138    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:46:32.121892 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.121852    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"]
Nov 01 03:46:32.122292 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.122197    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ec6bcc76-af29-4412-9477-2399fd51b1fb" containerName="util"
Nov 01 03:46:32.122292 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.122210    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6bcc76-af29-4412-9477-2399fd51b1fb" containerName="util"
Nov 01 03:46:32.122292 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.122230    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ec6bcc76-af29-4412-9477-2399fd51b1fb" containerName="extract"
Nov 01 03:46:32.122292 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.122235    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6bcc76-af29-4412-9477-2399fd51b1fb" containerName="extract"
Nov 01 03:46:32.122292 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.122255    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ec6bcc76-af29-4412-9477-2399fd51b1fb" containerName="pull"
Nov 01 03:46:32.122292 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.122259    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6bcc76-af29-4412-9477-2399fd51b1fb" containerName="pull"
Nov 01 03:46:32.122515 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.122310    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6bcc76-af29-4412-9477-2399fd51b1fb" containerName="extract"
Nov 01 03:46:32.125438 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.125409    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.223168 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.223135    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc2576c3-b8dd-4111-a9ce-b305e1965567-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.223335 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.223194    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc2576c3-b8dd-4111-a9ce-b305e1965567-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.223335 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.223251    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc2576c3-b8dd-4111-a9ce-b305e1965567-manifests\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.223335 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.223293    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc2576c3-b8dd-4111-a9ce-b305e1965567-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.223335 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.223309    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc2576c3-b8dd-4111-a9ce-b305e1965567-tmp\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.223501 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.223413    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p4nq\" (UniqueName: \"kubernetes.io/projected/dc2576c3-b8dd-4111-a9ce-b305e1965567-kube-api-access-9p4nq\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.260172 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.260137    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 03:46:32.260472 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.260421    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 03:46:32.260647 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.260498    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 03:46:32.265412 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.265391    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 03:46:32.265863 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.265745    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 03:46:32.305500 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.305468    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-t7bc9\""
Nov 01 03:46:32.323912 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.323885    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc2576c3-b8dd-4111-a9ce-b305e1965567-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.324009 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.323916    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc2576c3-b8dd-4111-a9ce-b305e1965567-manifests\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.324009 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.323953    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc2576c3-b8dd-4111-a9ce-b305e1965567-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.324009 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.323976    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc2576c3-b8dd-4111-a9ce-b305e1965567-tmp\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.324120 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.324021    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p4nq\" (UniqueName: \"kubernetes.io/projected/dc2576c3-b8dd-4111-a9ce-b305e1965567-kube-api-access-9p4nq\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.324120 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.324065    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc2576c3-b8dd-4111-a9ce-b305e1965567-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.324373 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.324349    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc2576c3-b8dd-4111-a9ce-b305e1965567-manifests\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.326332 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.326306    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc2576c3-b8dd-4111-a9ce-b305e1965567-tmp\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.326495 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.326475    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc2576c3-b8dd-4111-a9ce-b305e1965567-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.326821 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.326775    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc2576c3-b8dd-4111-a9ce-b305e1965567-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.326821 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.326799    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc2576c3-b8dd-4111-a9ce-b305e1965567-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.371087 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.371056    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"]
Nov 01 03:46:32.388665 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.388597    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p4nq\" (UniqueName: \"kubernetes.io/projected/dc2576c3-b8dd-4111-a9ce-b305e1965567-kube-api-access-9p4nq\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6\" (UID: \"dc2576c3-b8dd-4111-a9ce-b305e1965567\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.440405 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.440375    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:32.937540 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:32.937508    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"]
Nov 01 03:46:32.946490 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:46:32.946460    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2576c3_b8dd_4111_a9ce_b305e1965567.slice/crio-3d725a11b405dee8df1b623bd4c124d2ca2a3e2790d8668a53b2281f996e35fe WatchSource:0}: Error finding container 3d725a11b405dee8df1b623bd4c124d2ca2a3e2790d8668a53b2281f996e35fe: Status 404 returned error can't find the container with id 3d725a11b405dee8df1b623bd4c124d2ca2a3e2790d8668a53b2281f996e35fe
Nov 01 03:46:33.948252 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:33.948217    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6" event={"ID":"dc2576c3-b8dd-4111-a9ce-b305e1965567","Type":"ContainerStarted","Data":"3d725a11b405dee8df1b623bd4c124d2ca2a3e2790d8668a53b2281f996e35fe"}
Nov 01 03:46:38.974505 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:38.974469    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6" event={"ID":"dc2576c3-b8dd-4111-a9ce-b305e1965567","Type":"ContainerStarted","Data":"0c8db55311c85384f9b1887227f8a477c257e3e0982d2614267d95053f9e471a"}
Nov 01 03:46:39.981177 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:39.981145    4882 generic.go:356] "Generic (PLEG): container finished" podID="dc2576c3-b8dd-4111-a9ce-b305e1965567" containerID="0c8db55311c85384f9b1887227f8a477c257e3e0982d2614267d95053f9e471a" exitCode=0
Nov 01 03:46:39.981594 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:39.981195    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6" event={"ID":"dc2576c3-b8dd-4111-a9ce-b305e1965567","Type":"ContainerDied","Data":"0c8db55311c85384f9b1887227f8a477c257e3e0982d2614267d95053f9e471a"}
Nov 01 03:46:40.987909 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:40.987872    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6" event={"ID":"dc2576c3-b8dd-4111-a9ce-b305e1965567","Type":"ContainerStarted","Data":"1ef0ed2926eba054f327b6dfdc75cc4a79f931f057e51101d52f43073fc42669"}
Nov 01 03:46:40.988314 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:40.987983    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:41.209267 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:41.209217    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6" podStartSLOduration=3.293457313 podStartE2EDuration="10.209203535s" totalImagesPullingTime="6.498353724s" totalInitContainerRuntime="417.392498ms" isStatefulPod=false podCreationTimestamp="2026-11-01 03:46:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:46:41.16293788 +0000 UTC m=+269.724970909" watchObservedRunningTime="2026-11-01 03:46:41.209203535 +0000 UTC m=+269.771236553"
Nov 01 03:46:48.867353 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:48.867312    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp"]
Nov 01 03:46:48.874095 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:48.874066    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp"
Nov 01 03:46:48.925590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:48.925562    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 03:46:48.925721 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:48.925587    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 03:46:48.925721 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:48.925619    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wxvl5\""
Nov 01 03:46:48.968283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:48.968254    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpnln\" (UniqueName: \"kubernetes.io/projected/41c7a6e2-4a4f-44cb-9c73-b41108c72e30-kube-api-access-hpnln\") pod \"authorino-operator-7b7bb4bc77-g8ngp\" (UID: \"41c7a6e2-4a4f-44cb-9c73-b41108c72e30\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp"
Nov 01 03:46:48.973109 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:48.973086    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp"]
Nov 01 03:46:49.069172 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:49.069140    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpnln\" (UniqueName: \"kubernetes.io/projected/41c7a6e2-4a4f-44cb-9c73-b41108c72e30-kube-api-access-hpnln\") pod \"authorino-operator-7b7bb4bc77-g8ngp\" (UID: \"41c7a6e2-4a4f-44cb-9c73-b41108c72e30\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp"
Nov 01 03:46:49.126360 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:49.126296    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpnln\" (UniqueName: \"kubernetes.io/projected/41c7a6e2-4a4f-44cb-9c73-b41108c72e30-kube-api-access-hpnln\") pod \"authorino-operator-7b7bb4bc77-g8ngp\" (UID: \"41c7a6e2-4a4f-44cb-9c73-b41108c72e30\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp"
Nov 01 03:46:49.186395 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:49.186367    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp"
Nov 01 03:46:49.560061 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:49.560028    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp"]
Nov 01 03:46:49.568811 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:46:49.568787    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c7a6e2_4a4f_44cb_9c73_b41108c72e30.slice/crio-cde5d11f0df3bb22ac4bec88faa0672bcee766b4c9e19a2a83c0e964c26537a1 WatchSource:0}: Error finding container cde5d11f0df3bb22ac4bec88faa0672bcee766b4c9e19a2a83c0e964c26537a1: Status 404 returned error can't find the container with id cde5d11f0df3bb22ac4bec88faa0672bcee766b4c9e19a2a83c0e964c26537a1
Nov 01 03:46:50.027295 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:50.027262    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp" event={"ID":"41c7a6e2-4a4f-44cb-9c73-b41108c72e30","Type":"ContainerStarted","Data":"cde5d11f0df3bb22ac4bec88faa0672bcee766b4c9e19a2a83c0e964c26537a1"}
Nov 01 03:46:51.994448 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:51.994400    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-qc7z6"
Nov 01 03:46:55.052101 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:55.052059    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp" event={"ID":"41c7a6e2-4a4f-44cb-9c73-b41108c72e30","Type":"ContainerStarted","Data":"0c760c2c17075690edd20525c7227499ea139eac0d958036062fca39bb5e018c"}
Nov 01 03:46:55.052522 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:55.052180    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp"
Nov 01 03:46:55.163198 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:55.163138    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp" podStartSLOduration=2.365588124 podStartE2EDuration="7.163121846s" totalImagesPullingTime="4.797533722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:46:55.162336792 +0000 UTC m=+283.724369808" watchObservedRunningTime="2026-11-01 03:46:55.163121846 +0000 UTC m=+283.725154864"
Nov 01 03:46:57.335115 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:57.335079    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5"]
Nov 01 03:46:57.338094 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:57.338073    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5"
Nov 01 03:46:57.432870 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:57.432837    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 01 03:46:57.433026 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:57.432992    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bx4gw\""
Nov 01 03:46:57.434479 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:57.434459    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5"]
Nov 01 03:46:57.448125 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:57.448098    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbwj7\" (UniqueName: \"kubernetes.io/projected/06243225-e7c0-41c7-b49d-4bfa4b423d16-kube-api-access-wbwj7\") pod \"dns-operator-controller-manager-8557cbdb84-gd4v5\" (UID: \"06243225-e7c0-41c7-b49d-4bfa4b423d16\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5"
Nov 01 03:46:57.549004 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:57.548969    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbwj7\" (UniqueName: \"kubernetes.io/projected/06243225-e7c0-41c7-b49d-4bfa4b423d16-kube-api-access-wbwj7\") pod \"dns-operator-controller-manager-8557cbdb84-gd4v5\" (UID: \"06243225-e7c0-41c7-b49d-4bfa4b423d16\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5"
Nov 01 03:46:57.606205 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:57.606132    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbwj7\" (UniqueName: \"kubernetes.io/projected/06243225-e7c0-41c7-b49d-4bfa4b423d16-kube-api-access-wbwj7\") pod \"dns-operator-controller-manager-8557cbdb84-gd4v5\" (UID: \"06243225-e7c0-41c7-b49d-4bfa4b423d16\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5"
Nov 01 03:46:57.649937 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:57.649896    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5"
Nov 01 03:46:58.020402 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:58.020369    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5"]
Nov 01 03:46:58.028693 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:46:58.028666    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06243225_e7c0_41c7_b49d_4bfa4b423d16.slice/crio-131b11e64b51eb659fadd39cad6c9e128967457b8c9541f7db1b2475d2ae1082 WatchSource:0}: Error finding container 131b11e64b51eb659fadd39cad6c9e128967457b8c9541f7db1b2475d2ae1082: Status 404 returned error can't find the container with id 131b11e64b51eb659fadd39cad6c9e128967457b8c9541f7db1b2475d2ae1082
Nov 01 03:46:58.065347 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:46:58.065318    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5" event={"ID":"06243225-e7c0-41c7-b49d-4bfa4b423d16","Type":"ContainerStarted","Data":"131b11e64b51eb659fadd39cad6c9e128967457b8c9541f7db1b2475d2ae1082"}
Nov 01 03:46:58.724958 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:46:58.724923    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:47:03.088549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.088457    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5" event={"ID":"06243225-e7c0-41c7-b49d-4bfa4b423d16","Type":"ContainerStarted","Data":"abe52a2bf29fbb301a3197e85e170c38a1c7b92b0038c2061be46827dc839c3b"}
Nov 01 03:47:03.088908 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.088631    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5"
Nov 01 03:47:03.113549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.113516    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn"]
Nov 01 03:47:03.117104 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.117083    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn"
Nov 01 03:47:03.182940 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.182908    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 01 03:47:03.183238 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.183218    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 01 03:47:03.183238 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.183231    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-j444l\""
Nov 01 03:47:03.185866 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.185839    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn"]
Nov 01 03:47:03.197698 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.197671    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7kwj\" (UniqueName: \"kubernetes.io/projected/024c2154-ec29-4061-abda-09c304da8377-kube-api-access-t7kwj\") pod \"cert-manager-cainjector-5c57bbd4fc-g4wbn\" (UID: \"024c2154-ec29-4061-abda-09c304da8377\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn"
Nov 01 03:47:03.197862 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.197834    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/024c2154-ec29-4061-abda-09c304da8377-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-g4wbn\" (UID: \"024c2154-ec29-4061-abda-09c304da8377\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn"
Nov 01 03:47:03.252677 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.252611    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5" podStartSLOduration=1.546823323 podStartE2EDuration="6.252592718s" totalImagesPullingTime="4.705769395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:03.25142207 +0000 UTC m=+291.813455086" watchObservedRunningTime="2026-11-01 03:47:03.252592718 +0000 UTC m=+291.814625736"
Nov 01 03:47:03.298611 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.298570    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/024c2154-ec29-4061-abda-09c304da8377-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-g4wbn\" (UID: \"024c2154-ec29-4061-abda-09c304da8377\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn"
Nov 01 03:47:03.298774 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.298633    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kwj\" (UniqueName: \"kubernetes.io/projected/024c2154-ec29-4061-abda-09c304da8377-kube-api-access-t7kwj\") pod \"cert-manager-cainjector-5c57bbd4fc-g4wbn\" (UID: \"024c2154-ec29-4061-abda-09c304da8377\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn"
Nov 01 03:47:03.356470 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.356378    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/024c2154-ec29-4061-abda-09c304da8377-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-g4wbn\" (UID: \"024c2154-ec29-4061-abda-09c304da8377\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn"
Nov 01 03:47:03.356683 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.356664    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kwj\" (UniqueName: \"kubernetes.io/projected/024c2154-ec29-4061-abda-09c304da8377-kube-api-access-t7kwj\") pod \"cert-manager-cainjector-5c57bbd4fc-g4wbn\" (UID: \"024c2154-ec29-4061-abda-09c304da8377\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn"
Nov 01 03:47:03.444044 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.444013    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn"
Nov 01 03:47:03.818275 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:03.818236    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn"]
Nov 01 03:47:03.827040 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:47:03.827013    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod024c2154_ec29_4061_abda_09c304da8377.slice/crio-b4dd7734301815b3948e811e4cd424d771c4e9ced8411becae1abfbd741c5dba WatchSource:0}: Error finding container b4dd7734301815b3948e811e4cd424d771c4e9ced8411becae1abfbd741c5dba: Status 404 returned error can't find the container with id b4dd7734301815b3948e811e4cd424d771c4e9ced8411becae1abfbd741c5dba
Nov 01 03:47:04.094541 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:04.094419    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn" event={"ID":"024c2154-ec29-4061-abda-09c304da8377","Type":"ContainerStarted","Data":"b4dd7734301815b3948e811e4cd424d771c4e9ced8411becae1abfbd741c5dba"}
Nov 01 03:47:04.858567 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:04.858534    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"]
Nov 01 03:47:04.863112 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:04.863093    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:04.927723 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:04.927673    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"]
Nov 01 03:47:05.002916 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.002889    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xrhz5\""
Nov 01 03:47:05.010382 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.010354    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6csd\" (UniqueName: \"kubernetes.io/projected/8da28ad3-267f-43d9-9c13-9981e3cce024-kube-api-access-z6csd\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.010502 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.010399    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.010502 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.010438    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.010574 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.010511    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.010574 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.010563    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.010720 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.010705    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da28ad3-267f-43d9-9c13-9981e3cce024-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.010758 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.010737    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.010798 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.010773    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.010829 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.010817    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.111977 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.111902    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da28ad3-267f-43d9-9c13-9981e3cce024-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.111977 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.111937    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.111977 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.111961    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.112623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.111992    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.112623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.112020    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6csd\" (UniqueName: \"kubernetes.io/projected/8da28ad3-267f-43d9-9c13-9981e3cce024-kube-api-access-z6csd\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.112623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.112045    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.112623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.112064    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.112623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.112404    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.112623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.112450    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.112623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.112556    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.112623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.112569    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.112623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.112594    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-data\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.113002 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.112630    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.113002 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.112894    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da28ad3-267f-43d9-9c13-9981e3cce024-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.114323 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.114303    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.114490 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.114476    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.174248 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.174211    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6csd\" (UniqueName: \"kubernetes.io/projected/8da28ad3-267f-43d9-9c13-9981e3cce024-kube-api-access-z6csd\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.175531 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.175507    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da28ad3-267f-43d9-9c13-9981e3cce024-istio-token\") pod \"data-science-gateway-data-science-gateway-class-56c7694f85sv5q4\" (UID: \"8da28ad3-267f-43d9-9c13-9981e3cce024\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.476128 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.476094    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:05.925574 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.925478    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"]
Nov 01 03:47:05.934943 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:47:05.934911    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da28ad3_267f_43d9_9c13_9981e3cce024.slice/crio-bd590a2ecc7bd94cbd48f02d2c942a3249ad097a18efb0cf3538c5f7a84a9736 WatchSource:0}: Error finding container bd590a2ecc7bd94cbd48f02d2c942a3249ad097a18efb0cf3538c5f7a84a9736: Status 404 returned error can't find the container with id bd590a2ecc7bd94cbd48f02d2c942a3249ad097a18efb0cf3538c5f7a84a9736
Nov 01 03:47:05.938602 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.938114    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:47:05.938602 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.938188    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:47:05.938602 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:05.938222    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:47:06.060164 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.059441    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-g8ngp"
Nov 01 03:47:06.125476 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.113845    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4" event={"ID":"8da28ad3-267f-43d9-9c13-9981e3cce024","Type":"ContainerStarted","Data":"ddb7f60ca461a3a32ebc257e6205b57e079a8b4748351aa4825baac56ffb1cb2"}
Nov 01 03:47:06.125476 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.113896    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4" event={"ID":"8da28ad3-267f-43d9-9c13-9981e3cce024","Type":"ContainerStarted","Data":"bd590a2ecc7bd94cbd48f02d2c942a3249ad097a18efb0cf3538c5f7a84a9736"}
Nov 01 03:47:06.301013 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.296995    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4" podStartSLOduration=2.296976549 podStartE2EDuration="2.296976549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:06.295020515 +0000 UTC m=+294.857053532" watchObservedRunningTime="2026-11-01 03:47:06.296976549 +0000 UTC m=+294.859009569"
Nov 01 03:47:06.453875 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.453835    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"]
Nov 01 03:47:06.462669 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.462641    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:06.476842 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.476819    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:06.511825 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.511792    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z76k2\""
Nov 01 03:47:06.511944 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.511876    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 03:47:06.512372 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.512351    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"]
Nov 01 03:47:06.630540 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.630267    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpgwx\" (UniqueName: \"kubernetes.io/projected/7290de15-7ab7-446a-9d1d-854d9e394406-kube-api-access-mpgwx\") pod \"limitador-limitador-74dfcfb894-lcgvh\" (UID: \"7290de15-7ab7-446a-9d1d-854d9e394406\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:06.630540 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.630316    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7290de15-7ab7-446a-9d1d-854d9e394406-config-file\") pod \"limitador-limitador-74dfcfb894-lcgvh\" (UID: \"7290de15-7ab7-446a-9d1d-854d9e394406\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:06.714115 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.714033    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"]
Nov 01 03:47:06.731845 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.731806    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpgwx\" (UniqueName: \"kubernetes.io/projected/7290de15-7ab7-446a-9d1d-854d9e394406-kube-api-access-mpgwx\") pod \"limitador-limitador-74dfcfb894-lcgvh\" (UID: \"7290de15-7ab7-446a-9d1d-854d9e394406\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:06.732000 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.731861    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7290de15-7ab7-446a-9d1d-854d9e394406-config-file\") pod \"limitador-limitador-74dfcfb894-lcgvh\" (UID: \"7290de15-7ab7-446a-9d1d-854d9e394406\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:06.732575 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.732547    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7290de15-7ab7-446a-9d1d-854d9e394406-config-file\") pod \"limitador-limitador-74dfcfb894-lcgvh\" (UID: \"7290de15-7ab7-446a-9d1d-854d9e394406\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:06.800352 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:06.800307    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpgwx\" (UniqueName: \"kubernetes.io/projected/7290de15-7ab7-446a-9d1d-854d9e394406-kube-api-access-mpgwx\") pod \"limitador-limitador-74dfcfb894-lcgvh\" (UID: \"7290de15-7ab7-446a-9d1d-854d9e394406\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:07.081728 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:07.081692    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:07.468436 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:07.468389    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"]
Nov 01 03:47:07.477471 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:47:07.477422    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7290de15_7ab7_446a_9d1d_854d9e394406.slice/crio-609e2104aaa62792f26af9916671870b5f02f3d5e18ed17c8c7de3c844df854e WatchSource:0}: Error finding container 609e2104aaa62792f26af9916671870b5f02f3d5e18ed17c8c7de3c844df854e: Status 404 returned error can't find the container with id 609e2104aaa62792f26af9916671870b5f02f3d5e18ed17c8c7de3c844df854e
Nov 01 03:47:07.483251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:07.483220    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:07.898295 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:07.896662    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z52fh"]
Nov 01 03:47:07.902048 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:07.902023    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z52fh"
Nov 01 03:47:07.951917 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:07.951885    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mz5bh\""
Nov 01 03:47:07.954189 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:07.954163    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z52fh"]
Nov 01 03:47:08.044862 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:08.044828    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7sl6\" (UniqueName: \"kubernetes.io/projected/3c0ff0ef-3e5f-4226-b525-27e685ecff8c-kube-api-access-p7sl6\") pod \"authorino-57f7f8659f-z52fh\" (UID: \"3c0ff0ef-3e5f-4226-b525-27e685ecff8c\") " pod="kuadrant-system/authorino-57f7f8659f-z52fh"
Nov 01 03:47:08.138514 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:08.138485    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh" event={"ID":"7290de15-7ab7-446a-9d1d-854d9e394406","Type":"ContainerStarted","Data":"609e2104aaa62792f26af9916671870b5f02f3d5e18ed17c8c7de3c844df854e"}
Nov 01 03:47:08.140045 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:08.140017    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn" event={"ID":"024c2154-ec29-4061-abda-09c304da8377","Type":"ContainerStarted","Data":"e613e2d4d476acb7b96038b346a854138be521d916d836b5b690105464bc30ff"}
Nov 01 03:47:08.140325 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:08.140305    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:08.141457 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:08.141099    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-56c7694f85sv5q4"
Nov 01 03:47:08.145453 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:08.145418    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7sl6\" (UniqueName: \"kubernetes.io/projected/3c0ff0ef-3e5f-4226-b525-27e685ecff8c-kube-api-access-p7sl6\") pod \"authorino-57f7f8659f-z52fh\" (UID: \"3c0ff0ef-3e5f-4226-b525-27e685ecff8c\") " pod="kuadrant-system/authorino-57f7f8659f-z52fh"
Nov 01 03:47:08.202597 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:08.202570    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7sl6\" (UniqueName: \"kubernetes.io/projected/3c0ff0ef-3e5f-4226-b525-27e685ecff8c-kube-api-access-p7sl6\") pod \"authorino-57f7f8659f-z52fh\" (UID: \"3c0ff0ef-3e5f-4226-b525-27e685ecff8c\") " pod="kuadrant-system/authorino-57f7f8659f-z52fh"
Nov 01 03:47:08.216316 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:08.216284    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z52fh"
Nov 01 03:47:08.251382 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:08.251329    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g4wbn" podStartSLOduration=1.011763709 podStartE2EDuration="5.251316446s" totalImagesPullingTime="4.239552737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:08.250747332 +0000 UTC m=+296.812780348" watchObservedRunningTime="2026-11-01 03:47:08.251316446 +0000 UTC m=+296.813349462"
Nov 01 03:47:08.583814 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:08.583739    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z52fh"]
Nov 01 03:47:08.591988 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:47:08.591958    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c0ff0ef_3e5f_4226_b525_27e685ecff8c.slice/crio-4bed72ee9897d7aae294a70adb4e57736e5070e0d2d7c7e47faf8e5c95eccafc WatchSource:0}: Error finding container 4bed72ee9897d7aae294a70adb4e57736e5070e0d2d7c7e47faf8e5c95eccafc: Status 404 returned error can't find the container with id 4bed72ee9897d7aae294a70adb4e57736e5070e0d2d7c7e47faf8e5c95eccafc
Nov 01 03:47:09.145870 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:09.145826    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z52fh" event={"ID":"3c0ff0ef-3e5f-4226-b525-27e685ecff8c","Type":"ContainerStarted","Data":"4bed72ee9897d7aae294a70adb4e57736e5070e0d2d7c7e47faf8e5c95eccafc"}
Nov 01 03:47:11.156859 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:11.156820    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh" event={"ID":"7290de15-7ab7-446a-9d1d-854d9e394406","Type":"ContainerStarted","Data":"03f3740834153b72a0f11f2f6e8e0e3cf338cc428cdbf71fb8dc9dc860e8444a"}
Nov 01 03:47:11.157363 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:11.156987    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:11.266284 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:11.266230    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh" podStartSLOduration=2.516402993 podStartE2EDuration="5.266217238s" totalImagesPullingTime="2.749814245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:11.266031148 +0000 UTC m=+299.828064165" watchObservedRunningTime="2026-11-01 03:47:11.266217238 +0000 UTC m=+299.828250254"
Nov 01 03:47:11.619757 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:11.619730    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:47:11.619942 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:11.619780    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:47:12.163131 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:12.163098    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z52fh" event={"ID":"3c0ff0ef-3e5f-4226-b525-27e685ecff8c","Type":"ContainerStarted","Data":"c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0"}
Nov 01 03:47:12.272799 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:12.272745    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-z52fh" podStartSLOduration=2.090838824 podStartE2EDuration="5.272730834s" totalImagesPullingTime="3.18189201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:12.27097672 +0000 UTC m=+300.833009736" watchObservedRunningTime="2026-11-01 03:47:12.272730834 +0000 UTC m=+300.834763850"
Nov 01 03:47:14.097025 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:14.096996    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gd4v5"
Nov 01 03:47:14.601018 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:14.600974    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z52fh"]
Nov 01 03:47:14.601211 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:14.601116    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-z52fh" podUID="3c0ff0ef-3e5f-4226-b525-27e685ecff8c" containerName="authorino" containerID="cri-o://c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0" gracePeriod=30
Nov 01 03:47:14.850016 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:14.849991    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z52fh"
Nov 01 03:47:15.008093 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.008060    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0ff0ef-3e5f-4226-b525-27e685ecff8c-kube-api-access-p7sl6\" (UniqueName: \"kubernetes.io/projected/3c0ff0ef-3e5f-4226-b525-27e685ecff8c-kube-api-access-p7sl6\") pod \"3c0ff0ef-3e5f-4226-b525-27e685ecff8c\" (UID: \"3c0ff0ef-3e5f-4226-b525-27e685ecff8c\") "
Nov 01 03:47:15.010323 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.010295    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0ff0ef-3e5f-4226-b525-27e685ecff8c-kube-api-access-p7sl6" pod "3c0ff0ef-3e5f-4226-b525-27e685ecff8c" (UID: "3c0ff0ef-3e5f-4226-b525-27e685ecff8c"). InnerVolumeSpecName "kube-api-access-p7sl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:15.109083 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.109049    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7sl6\" (UniqueName: \"kubernetes.io/projected/3c0ff0ef-3e5f-4226-b525-27e685ecff8c-kube-api-access-p7sl6\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:47:15.179921 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.179881    4882 generic.go:356] "Generic (PLEG): container finished" podID="3c0ff0ef-3e5f-4226-b525-27e685ecff8c" containerID="c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0" exitCode=0
Nov 01 03:47:15.180073 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.179924    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z52fh"
Nov 01 03:47:15.180073 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.179973    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z52fh" event={"ID":"3c0ff0ef-3e5f-4226-b525-27e685ecff8c","Type":"ContainerDied","Data":"c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0"}
Nov 01 03:47:15.180073 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.180019    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z52fh" event={"ID":"3c0ff0ef-3e5f-4226-b525-27e685ecff8c","Type":"ContainerDied","Data":"4bed72ee9897d7aae294a70adb4e57736e5070e0d2d7c7e47faf8e5c95eccafc"}
Nov 01 03:47:15.180073 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.180039    4882 scope.go:122] "RemoveContainer" containerID="c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0"
Nov 01 03:47:15.189068 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.189049    4882 scope.go:122] "RemoveContainer" containerID="c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0"
Nov 01 03:47:15.189325 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:47:15.189305    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0\": container with ID starting with c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0 not found: ID does not exist" containerID="c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0"
Nov 01 03:47:15.189374 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.189333    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0"} err="failed to get container status \"c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0\": rpc error: code = NotFound desc = could not find container \"c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0\": container with ID starting with c5771926dc74a096ffa4b354339ebf003db4a10511a75f06be989e41d9e259d0 not found: ID does not exist"
Nov 01 03:47:15.331889 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.331817    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z52fh"]
Nov 01 03:47:15.338453 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.338415    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z52fh"]
Nov 01 03:47:15.597841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:15.597769    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0ff0ef-3e5f-4226-b525-27e685ecff8c" path="/var/lib/kubelet/pods/3c0ff0ef-3e5f-4226-b525-27e685ecff8c/volumes"
Nov 01 03:47:22.164318 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:22.164238    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:25.597945 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:25.597911    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"]
Nov 01 03:47:25.598469 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:25.598096    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh" podUID="7290de15-7ab7-446a-9d1d-854d9e394406" containerName="limitador" containerID="cri-o://03f3740834153b72a0f11f2f6e8e0e3cf338cc428cdbf71fb8dc9dc860e8444a" gracePeriod=30
Nov 01 03:47:26.231268 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:26.231233    4882 generic.go:356] "Generic (PLEG): container finished" podID="7290de15-7ab7-446a-9d1d-854d9e394406" containerID="03f3740834153b72a0f11f2f6e8e0e3cf338cc428cdbf71fb8dc9dc860e8444a" exitCode=0
Nov 01 03:47:26.231449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:26.231306    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh" event={"ID":"7290de15-7ab7-446a-9d1d-854d9e394406","Type":"ContainerDied","Data":"03f3740834153b72a0f11f2f6e8e0e3cf338cc428cdbf71fb8dc9dc860e8444a"}
Nov 01 03:47:26.545804 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:26.545780    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:26.604846 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:26.604817    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7290de15-7ab7-446a-9d1d-854d9e394406-config-file\" (UniqueName: \"kubernetes.io/configmap/7290de15-7ab7-446a-9d1d-854d9e394406-config-file\") pod \"7290de15-7ab7-446a-9d1d-854d9e394406\" (UID: \"7290de15-7ab7-446a-9d1d-854d9e394406\") "
Nov 01 03:47:26.605228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:26.604858    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7290de15-7ab7-446a-9d1d-854d9e394406-kube-api-access-mpgwx\" (UniqueName: \"kubernetes.io/projected/7290de15-7ab7-446a-9d1d-854d9e394406-kube-api-access-mpgwx\") pod \"7290de15-7ab7-446a-9d1d-854d9e394406\" (UID: \"7290de15-7ab7-446a-9d1d-854d9e394406\") "
Nov 01 03:47:26.605281 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:26.605219    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7290de15-7ab7-446a-9d1d-854d9e394406-config-file" pod "7290de15-7ab7-446a-9d1d-854d9e394406" (UID: "7290de15-7ab7-446a-9d1d-854d9e394406"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:47:26.606908 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:26.606886    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7290de15-7ab7-446a-9d1d-854d9e394406-kube-api-access-mpgwx" pod "7290de15-7ab7-446a-9d1d-854d9e394406" (UID: "7290de15-7ab7-446a-9d1d-854d9e394406"). InnerVolumeSpecName "kube-api-access-mpgwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:26.706096 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:26.706061    4882 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7290de15-7ab7-446a-9d1d-854d9e394406-config-file\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:47:26.706096 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:26.706091    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpgwx\" (UniqueName: \"kubernetes.io/projected/7290de15-7ab7-446a-9d1d-854d9e394406-kube-api-access-mpgwx\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:47:27.236825 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:27.236793    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"
Nov 01 03:47:27.236825 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:27.236809    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lcgvh" event={"ID":"7290de15-7ab7-446a-9d1d-854d9e394406","Type":"ContainerDied","Data":"609e2104aaa62792f26af9916671870b5f02f3d5e18ed17c8c7de3c844df854e"}
Nov 01 03:47:27.237075 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:27.236865    4882 scope.go:122] "RemoveContainer" containerID="03f3740834153b72a0f11f2f6e8e0e3cf338cc428cdbf71fb8dc9dc860e8444a"
Nov 01 03:47:27.388953 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:27.388919    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"]
Nov 01 03:47:27.435093 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:27.435060    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lcgvh"]
Nov 01 03:47:27.598284 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:27.598212    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7290de15-7ab7-446a-9d1d-854d9e394406" path="/var/lib/kubelet/pods/7290de15-7ab7-446a-9d1d-854d9e394406/volumes"
Nov 01 03:47:28.722497 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:47:28.722458    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:47:29.637141 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.637111    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"]
Nov 01 03:47:29.637509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.637495    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3c0ff0ef-3e5f-4226-b525-27e685ecff8c" containerName="authorino"
Nov 01 03:47:29.637566 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.637509    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0ff0ef-3e5f-4226-b525-27e685ecff8c" containerName="authorino"
Nov 01 03:47:29.637566 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.637524    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7290de15-7ab7-446a-9d1d-854d9e394406" containerName="limitador"
Nov 01 03:47:29.637566 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.637529    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7290de15-7ab7-446a-9d1d-854d9e394406" containerName="limitador"
Nov 01 03:47:29.637871 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.637609    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0ff0ef-3e5f-4226-b525-27e685ecff8c" containerName="authorino"
Nov 01 03:47:29.637871 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.637617    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7290de15-7ab7-446a-9d1d-854d9e394406" containerName="limitador"
Nov 01 03:47:29.645758 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.645738    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.695397 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.695356    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 03:47:29.695397 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.695385    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-kj6gp\""
Nov 01 03:47:29.695661 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.695582    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 03:47:29.697409 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.697383    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"]
Nov 01 03:47:29.733223 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.733195    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8km\" (UniqueName: \"kubernetes.io/projected/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-kube-api-access-lx8km\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.733571 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.733239    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.733571 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.733348    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.733571 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.733382    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.787929 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.787895    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 03:47:29.834815 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.834784    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8km\" (UniqueName: \"kubernetes.io/projected/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-kube-api-access-lx8km\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.834933 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.834821    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.834933 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.834854    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.834933 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.834874    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.835173 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.835154    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.835238 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.835184    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.835714 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.835697    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.892200 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.892147    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8km\" (UniqueName: \"kubernetes.io/projected/83ed9b69-e27e-451b-8ab9-2fc60f6621f9-kube-api-access-lx8km\") pod \"kserve-module-controller-manager-7566cf5fc8-7nrj5\" (UID: \"83ed9b69-e27e-451b-8ab9-2fc60f6621f9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:29.958141 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:29.958113    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:30.382574 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:30.382544    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"]
Nov 01 03:47:30.390853 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:47:30.390826    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ed9b69_e27e_451b_8ab9_2fc60f6621f9.slice/crio-3a11683dc113961f28478168a6234577bdd84ba48336f47ccef70212549856b5 WatchSource:0}: Error finding container 3a11683dc113961f28478168a6234577bdd84ba48336f47ccef70212549856b5: Status 404 returned error can't find the container with id 3a11683dc113961f28478168a6234577bdd84ba48336f47ccef70212549856b5
Nov 01 03:47:30.392890 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:30.392870    4882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:47:31.257346 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:31.257313    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5" event={"ID":"83ed9b69-e27e-451b-8ab9-2fc60f6621f9","Type":"ContainerStarted","Data":"3a11683dc113961f28478168a6234577bdd84ba48336f47ccef70212549856b5"}
Nov 01 03:47:36.282914 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:36.282877    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5" event={"ID":"83ed9b69-e27e-451b-8ab9-2fc60f6621f9","Type":"ContainerStarted","Data":"df1e1a8865aa952a11064099febeb677d99ae1e9a4a520d682adac83f8129992"}
Nov 01 03:47:36.283418 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:36.283011    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:36.441033 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:36.440981    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5" podStartSLOduration=2.586236553 podStartE2EDuration="7.440966041s" totalImagesPullingTime="4.854729488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:36.439809645 +0000 UTC m=+325.001842663" watchObservedRunningTime="2026-11-01 03:47:36.440966041 +0000 UTC m=+325.002999057"
Nov 01 03:47:46.301143 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.301108    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"]
Nov 01 03:47:46.309670 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.309651    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"
Nov 01 03:47:46.358457 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.358412    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:47:46.358643 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.358611    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:47:46.359503 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.359482    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"]
Nov 01 03:47:46.386727 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.386696    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn7kx\" (UniqueName: \"kubernetes.io/projected/24ec074e-6d28-431a-9327-25352b16dc0c-kube-api-access-tn7kx\") pod \"postgres-7cdbf6dc44-4qgp8\" (UID: \"24ec074e-6d28-431a-9327-25352b16dc0c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"
Nov 01 03:47:46.386850 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.386759    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/24ec074e-6d28-431a-9327-25352b16dc0c-data\") pod \"postgres-7cdbf6dc44-4qgp8\" (UID: \"24ec074e-6d28-431a-9327-25352b16dc0c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"
Nov 01 03:47:46.403301 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.403281    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 01 03:47:46.403301 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.403296    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vk5fp\""
Nov 01 03:47:46.488058 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.488022    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/24ec074e-6d28-431a-9327-25352b16dc0c-data\") pod \"postgres-7cdbf6dc44-4qgp8\" (UID: \"24ec074e-6d28-431a-9327-25352b16dc0c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"
Nov 01 03:47:46.488231 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.488096    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn7kx\" (UniqueName: \"kubernetes.io/projected/24ec074e-6d28-431a-9327-25352b16dc0c-kube-api-access-tn7kx\") pod \"postgres-7cdbf6dc44-4qgp8\" (UID: \"24ec074e-6d28-431a-9327-25352b16dc0c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"
Nov 01 03:47:46.488469 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.488450    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/24ec074e-6d28-431a-9327-25352b16dc0c-data\") pod \"postgres-7cdbf6dc44-4qgp8\" (UID: \"24ec074e-6d28-431a-9327-25352b16dc0c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"
Nov 01 03:47:46.545964 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.545931    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn7kx\" (UniqueName: \"kubernetes.io/projected/24ec074e-6d28-431a-9327-25352b16dc0c-kube-api-access-tn7kx\") pod \"postgres-7cdbf6dc44-4qgp8\" (UID: \"24ec074e-6d28-431a-9327-25352b16dc0c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"
Nov 01 03:47:46.623248 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.623168    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"
Nov 01 03:47:46.993248 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:46.993216    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"]
Nov 01 03:47:47.002173 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:47:47.002146    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ec074e_6d28_431a_9327_25352b16dc0c.slice/crio-3289565e7fcd05ee8e021bd8f6fefea61daf96bb1d588c35f4b60aaf5cda9474 WatchSource:0}: Error finding container 3289565e7fcd05ee8e021bd8f6fefea61daf96bb1d588c35f4b60aaf5cda9474: Status 404 returned error can't find the container with id 3289565e7fcd05ee8e021bd8f6fefea61daf96bb1d588c35f4b60aaf5cda9474
Nov 01 03:47:47.342239 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:47.342159    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8" event={"ID":"24ec074e-6d28-431a-9327-25352b16dc0c","Type":"ContainerStarted","Data":"3289565e7fcd05ee8e021bd8f6fefea61daf96bb1d588c35f4b60aaf5cda9474"}
Nov 01 03:47:57.289639 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:47:57.289600    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7nrj5"
Nov 01 03:47:58.724284 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:47:58.724243    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:48:02.421925 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:02.421873    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8" event={"ID":"24ec074e-6d28-431a-9327-25352b16dc0c","Type":"ContainerStarted","Data":"60980eee5fbf86ef9ce8808fcaca5f392f13c2a822cd3729378afd8dfb963271"}
Nov 01 03:48:02.422310 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:02.421945    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"
Nov 01 03:48:02.530029 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:02.529961    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8" podStartSLOduration=2.073795582 podStartE2EDuration="16.529942411s" totalImagesPullingTime="14.456146829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:48:02.529252732 +0000 UTC m=+351.091285749" watchObservedRunningTime="2026-11-01 03:48:02.529942411 +0000 UTC m=+351.091975428"
Nov 01 03:48:08.456337 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:08.456305    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4qgp8"
Nov 01 03:48:26.240942 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.240907    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"]
Nov 01 03:48:26.244396 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.244372    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.306626 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.306597    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 01 03:48:26.306626 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.306608    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 01 03:48:26.349348 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.349318    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/20a09652-c72e-4f0f-a539-0dba5964c838-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.349509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.349351    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20a09652-c72e-4f0f-a539-0dba5964c838-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.349509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.349379    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20a09652-c72e-4f0f-a539-0dba5964c838-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.349509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.349496    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rksr4\" (UniqueName: \"kubernetes.io/projected/20a09652-c72e-4f0f-a539-0dba5964c838-kube-api-access-rksr4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.353104 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.353080    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 01 03:48:26.353258 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.353246    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-pvfwr\""
Nov 01 03:48:26.355025 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.355004    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"]
Nov 01 03:48:26.450209 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.450175    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/20a09652-c72e-4f0f-a539-0dba5964c838-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.450419 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.450219    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20a09652-c72e-4f0f-a539-0dba5964c838-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.450419 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.450249    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20a09652-c72e-4f0f-a539-0dba5964c838-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.450419 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.450276    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksr4\" (UniqueName: \"kubernetes.io/projected/20a09652-c72e-4f0f-a539-0dba5964c838-kube-api-access-rksr4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.450419 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:48:26.450381    4882 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 01 03:48:26.450419 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:48:26.450402    4882 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 01 03:48:26.450764 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:48:26.450467    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a09652-c72e-4f0f-a539-0dba5964c838-tls-cert podName:20a09652-c72e-4f0f-a539-0dba5964c838 nodeName:}" failed. No retries permitted until 2026-11-01 03:48:26.950421927 +0000 UTC m=+375.512454945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/20a09652-c72e-4f0f-a539-0dba5964c838-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-lcsnr" (UID: "20a09652-c72e-4f0f-a539-0dba5964c838") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 01 03:48:26.450764 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:48:26.450496    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20a09652-c72e-4f0f-a539-0dba5964c838-proxy-source podName:20a09652-c72e-4f0f-a539-0dba5964c838 nodeName:}" failed. No retries permitted until 2026-11-01 03:48:26.95047478 +0000 UTC m=+375.512507787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/20a09652-c72e-4f0f-a539-0dba5964c838-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-lcsnr" (UID: "20a09652-c72e-4f0f-a539-0dba5964c838") : configmap "usage-tenancy-proxy-source" not found
Nov 01 03:48:26.451260 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.451236    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20a09652-c72e-4f0f-a539-0dba5964c838-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.513476 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.513381    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rksr4\" (UniqueName: \"kubernetes.io/projected/20a09652-c72e-4f0f-a539-0dba5964c838-kube-api-access-rksr4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.956276 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.956247    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/20a09652-c72e-4f0f-a539-0dba5964c838-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.956450 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.956282    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20a09652-c72e-4f0f-a539-0dba5964c838-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.957094 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.957073    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/20a09652-c72e-4f0f-a539-0dba5964c838-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:26.958701 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:26.958679    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20a09652-c72e-4f0f-a539-0dba5964c838-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lcsnr\" (UID: \"20a09652-c72e-4f0f-a539-0dba5964c838\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:27.157343 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:27.157313    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:27.532934 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:27.532848    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"]
Nov 01 03:48:27.541598 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:48:27.541569    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a09652_c72e_4f0f_a539_0dba5964c838.slice/crio-0a58eda09c79f2a116f7f34350579b4bef55f00fa57b7f1589270fb1bda9dca5 WatchSource:0}: Error finding container 0a58eda09c79f2a116f7f34350579b4bef55f00fa57b7f1589270fb1bda9dca5: Status 404 returned error can't find the container with id 0a58eda09c79f2a116f7f34350579b4bef55f00fa57b7f1589270fb1bda9dca5
Nov 01 03:48:27.551963 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:27.551937    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr" event={"ID":"20a09652-c72e-4f0f-a539-0dba5964c838","Type":"ContainerStarted","Data":"0a58eda09c79f2a116f7f34350579b4bef55f00fa57b7f1589270fb1bda9dca5"}
Nov 01 03:48:28.722930 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:48:28.722889    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:48:31.805043 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.805010    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"]
Nov 01 03:48:31.808763 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.808733    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:31.903224 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.903183    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7391880-8ea9-497a-bf39-a508a6227349-cert\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:31.903472 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.903248    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkfl4\" (UniqueName: \"kubernetes.io/projected/f7391880-8ea9-497a-bf39-a508a6227349-kube-api-access-lkfl4\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:31.903472 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.903333    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7391880-8ea9-497a-bf39-a508a6227349-metrics-certs\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:31.903472 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.903382    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7391880-8ea9-497a-bf39-a508a6227349-manager-config\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:31.903848 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.903824    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:48:31.903979 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.903848    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-476mj\""
Nov 01 03:48:31.903979 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.903881    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 03:48:31.906670 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.906645    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"]
Nov 01 03:48:31.958853 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.958816    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 01 03:48:31.959109 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.959092    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 01 03:48:31.959716 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:31.959701    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 01 03:48:32.004035 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:32.004002    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7391880-8ea9-497a-bf39-a508a6227349-cert\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:32.004189 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:32.004055    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkfl4\" (UniqueName: \"kubernetes.io/projected/f7391880-8ea9-497a-bf39-a508a6227349-kube-api-access-lkfl4\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:32.004189 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:32.004081    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7391880-8ea9-497a-bf39-a508a6227349-metrics-certs\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:32.004189 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:32.004122    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7391880-8ea9-497a-bf39-a508a6227349-manager-config\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:32.006019 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:32.005535    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7391880-8ea9-497a-bf39-a508a6227349-manager-config\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:32.007739 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:32.007714    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7391880-8ea9-497a-bf39-a508a6227349-metrics-certs\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:32.018516 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:32.018490    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7391880-8ea9-497a-bf39-a508a6227349-cert\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:32.062132 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:32.062062    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkfl4\" (UniqueName: \"kubernetes.io/projected/f7391880-8ea9-497a-bf39-a508a6227349-kube-api-access-lkfl4\") pod \"jobset-controller-manager-646cd44-v6vmr\" (UID: \"f7391880-8ea9-497a-bf39-a508a6227349\") " pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:32.126177 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:32.126132    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:32.545775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:32.545734    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"]
Nov 01 03:48:32.558971 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:48:32.558932    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7391880_8ea9_497a_bf39_a508a6227349.slice/crio-ef2bbefc28c95d3493a423b27e877cb7a059140e11a211e9ebd1adac918c8857 WatchSource:0}: Error finding container ef2bbefc28c95d3493a423b27e877cb7a059140e11a211e9ebd1adac918c8857: Status 404 returned error can't find the container with id ef2bbefc28c95d3493a423b27e877cb7a059140e11a211e9ebd1adac918c8857
Nov 01 03:48:32.583499 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:32.583466    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr" event={"ID":"f7391880-8ea9-497a-bf39-a508a6227349","Type":"ContainerStarted","Data":"ef2bbefc28c95d3493a423b27e877cb7a059140e11a211e9ebd1adac918c8857"}
Nov 01 03:48:37.615214 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:37.615170    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr" event={"ID":"f7391880-8ea9-497a-bf39-a508a6227349","Type":"ContainerStarted","Data":"f2d44b83e329c2012768e24438c560ccd033ef9cb9a73c893364a72e7d1dda4d"}
Nov 01 03:48:37.615659 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:37.615247    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:37.725093 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:37.725043    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr" podStartSLOduration=2.340192917 podStartE2EDuration="6.725028229s" totalImagesPullingTime="4.384835312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:48:37.723531842 +0000 UTC m=+386.285564856" watchObservedRunningTime="2026-11-01 03:48:37.725028229 +0000 UTC m=+386.287061246"
Nov 01 03:48:46.664696 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:46.664657    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr" event={"ID":"20a09652-c72e-4f0f-a539-0dba5964c838","Type":"ContainerStarted","Data":"9a1fa71ea9c085608f8e93d3a0c1c62f7837210c8e7da0ccc20e3ab7803daac3"}
Nov 01 03:48:46.665089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:46.664844    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:46.774441 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:46.774363    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr" podStartSLOduration=2.167917697 podStartE2EDuration="20.774350081s" totalImagesPullingTime="18.606432384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:48:46.773204583 +0000 UTC m=+395.335237600" watchObservedRunningTime="2026-11-01 03:48:46.774350081 +0000 UTC m=+395.336383127"
Nov 01 03:48:48.627974 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:48.627940    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-646cd44-v6vmr"
Nov 01 03:48:57.672622 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:48:57.672542    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lcsnr"
Nov 01 03:48:58.723385 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:48:58.723348    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:49:11.634591 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:11.634554    4882 scope.go:122] "RemoveContainer" containerID="fd4dd5d559c5421f9c558819e7e4ca68dbb33dadd9612dc1c03cb35ec86dbf7a"
Nov 01 03:49:13.839775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:13.820537    4882 generic.go:356] "Generic (PLEG): container finished" podID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerID="000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5" exitCode=0
Nov 01 03:49:13.839775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:13.820618    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" event={"ID":"732dba6f-5c7f-48bf-8802-6742d16cc51d","Type":"ContainerDied","Data":"000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5"}
Nov 01 03:49:13.839775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:13.820967    4882 scope.go:122] "RemoveContainer" containerID="000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5"
Nov 01 03:49:13.839775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:13.822048    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:49:13.839775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:13.822111    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:49:13.839775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:13.822142    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:49:14.657802 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:14.657763    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cc78785cc-cdrj4"]
Nov 01 03:49:14.703070 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:14.703044    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc78785cc-cdrj4"
Nov 01 03:49:14.714500 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:14.714421    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cc78785cc-cdrj4"]
Nov 01 03:49:14.752130 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:14.752092    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mz5bh\""
Nov 01 03:49:14.787649 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:14.787603    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:49:14.827844 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:14.827807    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" event={"ID":"732dba6f-5c7f-48bf-8802-6742d16cc51d","Type":"ContainerStarted","Data":"d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6"}
Nov 01 03:49:14.836576 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:14.836548    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkn66\" (UniqueName: \"kubernetes.io/projected/1820a50a-d463-40ad-8bc7-743865af8ea2-kube-api-access-rkn66\") pod \"authorino-7cc78785cc-cdrj4\" (UID: \"1820a50a-d463-40ad-8bc7-743865af8ea2\") " pod="kuadrant-system/authorino-7cc78785cc-cdrj4"
Nov 01 03:49:14.937801 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:14.937768    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkn66\" (UniqueName: \"kubernetes.io/projected/1820a50a-d463-40ad-8bc7-743865af8ea2-kube-api-access-rkn66\") pod \"authorino-7cc78785cc-cdrj4\" (UID: \"1820a50a-d463-40ad-8bc7-743865af8ea2\") " pod="kuadrant-system/authorino-7cc78785cc-cdrj4"
Nov 01 03:49:14.986521 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:14.986142    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cc78785cc-cdrj4"]
Nov 01 03:49:14.986521 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:49:14.986449    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rkn66], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7cc78785cc-cdrj4" podUID="1820a50a-d463-40ad-8bc7-743865af8ea2"
Nov 01 03:49:14.998156 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:14.998125    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkn66\" (UniqueName: \"kubernetes.io/projected/1820a50a-d463-40ad-8bc7-743865af8ea2-kube-api-access-rkn66\") pod \"authorino-7cc78785cc-cdrj4\" (UID: \"1820a50a-d463-40ad-8bc7-743865af8ea2\") " pod="kuadrant-system/authorino-7cc78785cc-cdrj4"
Nov 01 03:49:15.018715 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.018689    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-664b5b9867-x4lkz"]
Nov 01 03:49:15.059047 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.059019    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-664b5b9867-x4lkz"
Nov 01 03:49:15.084452 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.079709    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-664b5b9867-x4lkz"]
Nov 01 03:49:15.107976 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.107949    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 03:49:15.139053 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.139020    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/046a104e-7cbc-4530-99d9-7cab7717f09d-tls-cert\") pod \"authorino-664b5b9867-x4lkz\" (UID: \"046a104e-7cbc-4530-99d9-7cab7717f09d\") " pod="kuadrant-system/authorino-664b5b9867-x4lkz"
Nov 01 03:49:15.139184 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.139063    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld2rb\" (UniqueName: \"kubernetes.io/projected/046a104e-7cbc-4530-99d9-7cab7717f09d-kube-api-access-ld2rb\") pod \"authorino-664b5b9867-x4lkz\" (UID: \"046a104e-7cbc-4530-99d9-7cab7717f09d\") " pod="kuadrant-system/authorino-664b5b9867-x4lkz"
Nov 01 03:49:15.240240 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.240146    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/046a104e-7cbc-4530-99d9-7cab7717f09d-tls-cert\") pod \"authorino-664b5b9867-x4lkz\" (UID: \"046a104e-7cbc-4530-99d9-7cab7717f09d\") " pod="kuadrant-system/authorino-664b5b9867-x4lkz"
Nov 01 03:49:15.240240 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.240194    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld2rb\" (UniqueName: \"kubernetes.io/projected/046a104e-7cbc-4530-99d9-7cab7717f09d-kube-api-access-ld2rb\") pod \"authorino-664b5b9867-x4lkz\" (UID: \"046a104e-7cbc-4530-99d9-7cab7717f09d\") " pod="kuadrant-system/authorino-664b5b9867-x4lkz"
Nov 01 03:49:15.242630 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.242607    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/046a104e-7cbc-4530-99d9-7cab7717f09d-tls-cert\") pod \"authorino-664b5b9867-x4lkz\" (UID: \"046a104e-7cbc-4530-99d9-7cab7717f09d\") " pod="kuadrant-system/authorino-664b5b9867-x4lkz"
Nov 01 03:49:15.314274 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.314246    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld2rb\" (UniqueName: \"kubernetes.io/projected/046a104e-7cbc-4530-99d9-7cab7717f09d-kube-api-access-ld2rb\") pod \"authorino-664b5b9867-x4lkz\" (UID: \"046a104e-7cbc-4530-99d9-7cab7717f09d\") " pod="kuadrant-system/authorino-664b5b9867-x4lkz"
Nov 01 03:49:15.419333 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.419299    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-664b5b9867-x4lkz"
Nov 01 03:49:15.649968 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.649939    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:49:15.651029 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.650997    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vjrbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 03:49:15.651166 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.651044    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 03:49:15.836034 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.833586    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-664b5b9867-x4lkz"]
Nov 01 03:49:15.836034 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.835785    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc78785cc-cdrj4"
Nov 01 03:49:15.840728 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:49:15.840704    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod046a104e_7cbc_4530_99d9_7cab7717f09d.slice/crio-b32387db1a2b1e64d6efdfd78d9a3d517301d608550b4d3fad6feb7cea9a9603 WatchSource:0}: Error finding container b32387db1a2b1e64d6efdfd78d9a3d517301d608550b4d3fad6feb7cea9a9603: Status 404 returned error can't find the container with id b32387db1a2b1e64d6efdfd78d9a3d517301d608550b4d3fad6feb7cea9a9603
Nov 01 03:49:15.886625 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.886593    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc78785cc-cdrj4"
Nov 01 03:49:15.945843 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.945807    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1820a50a-d463-40ad-8bc7-743865af8ea2-kube-api-access-rkn66\" (UniqueName: \"kubernetes.io/projected/1820a50a-d463-40ad-8bc7-743865af8ea2-kube-api-access-rkn66\") pod \"1820a50a-d463-40ad-8bc7-743865af8ea2\" (UID: \"1820a50a-d463-40ad-8bc7-743865af8ea2\") "
Nov 01 03:49:15.948142 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:15.948113    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1820a50a-d463-40ad-8bc7-743865af8ea2-kube-api-access-rkn66" pod "1820a50a-d463-40ad-8bc7-743865af8ea2" (UID: "1820a50a-d463-40ad-8bc7-743865af8ea2"). InnerVolumeSpecName "kube-api-access-rkn66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:49:16.047175 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:16.047140    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkn66\" (UniqueName: \"kubernetes.io/projected/1820a50a-d463-40ad-8bc7-743865af8ea2-kube-api-access-rkn66\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:49:16.650332 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:16.650297    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vjrbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 03:49:16.650516 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:16.650362    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 03:49:16.841200 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:16.841158    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-664b5b9867-x4lkz" event={"ID":"046a104e-7cbc-4530-99d9-7cab7717f09d","Type":"ContainerStarted","Data":"b32387db1a2b1e64d6efdfd78d9a3d517301d608550b4d3fad6feb7cea9a9603"}
Nov 01 03:49:16.841368 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:16.841228    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc78785cc-cdrj4"
Nov 01 03:49:17.053626 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:17.053594    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cc78785cc-cdrj4"]
Nov 01 03:49:17.057516 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:17.057483    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7cc78785cc-cdrj4"]
Nov 01 03:49:17.598511 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:17.598478    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1820a50a-d463-40ad-8bc7-743865af8ea2" path="/var/lib/kubelet/pods/1820a50a-d463-40ad-8bc7-743865af8ea2/volumes"
Nov 01 03:49:17.649554 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:17.649521    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vjrbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 03:49:17.649704 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:17.649570    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 03:49:18.649757 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:18.649725    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vjrbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 03:49:18.650138 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:18.649821    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 03:49:18.852395 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:18.852361    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-664b5b9867-x4lkz" event={"ID":"046a104e-7cbc-4530-99d9-7cab7717f09d","Type":"ContainerStarted","Data":"276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d"}
Nov 01 03:49:19.649928 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:19.649890    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-vjrbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 03:49:19.650420 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:19.649961    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 03:49:20.653466 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:20.653424    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:49:20.653834 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:20.653701    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:49:20.654420 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:20.654400    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:49:20.814023 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:49:20.813969    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-664b5b9867-x4lkz" podStartSLOduration=4.868780675 podStartE2EDuration="6.813953526s" totalImagesPullingTime="1.945172851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:49:18.986232657 +0000 UTC m=+427.548265676" watchObservedRunningTime="2026-11-01 03:49:20.813953526 +0000 UTC m=+429.375986542"
Nov 01 03:49:28.722386 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:49:28.722347    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:49:58.722599 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:49:58.722561    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:50:28.722923 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:50:28.722843    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:50:58.721549 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:50:58.721512    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:51:24.878645 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:24.878611    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"]
Nov 01 03:51:24.882243 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:24.882220    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:24.930997 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:24.930958    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 03:51:24.976804 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:24.976778    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 03:51:24.976804 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:24.976806    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dpn7r\""
Nov 01 03:51:24.976982 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:24.976817    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 01 03:51:24.978734 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:24.978714    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"]
Nov 01 03:51:25.002463 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.002418    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.002587 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.002488    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39634861-5963-4ce5-abaa-8aafd19668a7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.002587 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.002566    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26gzw\" (UniqueName: \"kubernetes.io/projected/39634861-5963-4ce5-abaa-8aafd19668a7-kube-api-access-26gzw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.002680 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.002604    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.002680 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.002623    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.002758 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.002738    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.104253 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.104211    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.104467 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.104330    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.104467 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.104396    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39634861-5963-4ce5-abaa-8aafd19668a7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.104605 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.104476    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26gzw\" (UniqueName: \"kubernetes.io/projected/39634861-5963-4ce5-abaa-8aafd19668a7-kube-api-access-26gzw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.104605 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.104515    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.104605 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.104542    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.104805 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.104780    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.104865 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.104807    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.104953 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.104934    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.107438 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.107405    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39634861-5963-4ce5-abaa-8aafd19668a7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.107632 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.107614    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39634861-5963-4ce5-abaa-8aafd19668a7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.161706 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.161637    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26gzw\" (UniqueName: \"kubernetes.io/projected/39634861-5963-4ce5-abaa-8aafd19668a7-kube-api-access-26gzw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tzkq2\" (UID: \"39634861-5963-4ce5-abaa-8aafd19668a7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.194603 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.194571    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:25.571078 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:25.571037    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"]
Nov 01 03:51:25.579284 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:51:25.579257    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39634861_5963_4ce5_abaa_8aafd19668a7.slice/crio-d55b0fee495c8db5adbf82ce1a3e49863de827a68dc5978c72a29900ac77b9c2 WatchSource:0}: Error finding container d55b0fee495c8db5adbf82ce1a3e49863de827a68dc5978c72a29900ac77b9c2: Status 404 returned error can't find the container with id d55b0fee495c8db5adbf82ce1a3e49863de827a68dc5978c72a29900ac77b9c2
Nov 01 03:51:26.460518 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:26.460482    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2" event={"ID":"39634861-5963-4ce5-abaa-8aafd19668a7","Type":"ContainerStarted","Data":"d55b0fee495c8db5adbf82ce1a3e49863de827a68dc5978c72a29900ac77b9c2"}
Nov 01 03:51:27.466717 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.466675    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2" event={"ID":"39634861-5963-4ce5-abaa-8aafd19668a7","Type":"ContainerStarted","Data":"2ec5f8fe745a404d1f236d93d327c9720ec01e86bda4404b9349241607b4a508"}
Nov 01 03:51:27.579663 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.579606    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2" podStartSLOduration=2.081734381 podStartE2EDuration="3.57958669s" totalImagesPullingTime="1.497852309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:27.577774593 +0000 UTC m=+556.139807609" watchObservedRunningTime="2026-11-01 03:51:27.57958669 +0000 UTC m=+556.141619708"
Nov 01 03:51:27.822764 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.822683    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"]
Nov 01 03:51:27.826544 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.826512    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:27.877484 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.877454    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 01 03:51:27.882024 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.881991    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"]
Nov 01 03:51:27.934270 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.934232    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2std6\" (UniqueName: \"kubernetes.io/projected/9499604f-575b-4971-b182-1073437989c7-kube-api-access-2std6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:27.934405 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.934285    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:27.934405 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.934381    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9499604f-575b-4971-b182-1073437989c7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:27.934505 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.934410    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:27.934542 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.934516    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:27.934542 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:27.934537    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.035649 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.035617    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2std6\" (UniqueName: \"kubernetes.io/projected/9499604f-575b-4971-b182-1073437989c7-kube-api-access-2std6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.035649 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.035654    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.035867 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.035681    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9499604f-575b-4971-b182-1073437989c7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.035867 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.035697    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.035867 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.035745    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.035867 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.035763    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.036151 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.036120    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.036267 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.036135    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.036267 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.036187    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.037986 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.037967    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9499604f-575b-4971-b182-1073437989c7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.038168 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.038152    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9499604f-575b-4971-b182-1073437989c7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.099998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.099926    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2std6\" (UniqueName: \"kubernetes.io/projected/9499604f-575b-4971-b182-1073437989c7-kube-api-access-2std6\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6756s\" (UID: \"9499604f-575b-4971-b182-1073437989c7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.139910 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.139877    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:28.196784 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.196744    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:28.516246 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:28.516215    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"]
Nov 01 03:51:28.524378 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:51:28.524352    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9499604f_575b_4971_b182_1073437989c7.slice/crio-8cd7725bda9a350f1e07852ec4462eacd1fc19b25d689b338594e9d712327cd4 WatchSource:0}: Error finding container 8cd7725bda9a350f1e07852ec4462eacd1fc19b25d689b338594e9d712327cd4: Status 404 returned error can't find the container with id 8cd7725bda9a350f1e07852ec4462eacd1fc19b25d689b338594e9d712327cd4
Nov 01 03:51:28.722312 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:51:28.722279    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:51:29.478259 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:29.478165    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s" event={"ID":"9499604f-575b-4971-b182-1073437989c7","Type":"ContainerStarted","Data":"83eaacef95b0f475e06ea409fcaf80c29bab04078020f5fbd5f89592e5163085"}
Nov 01 03:51:29.478259 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:29.478233    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s" event={"ID":"9499604f-575b-4971-b182-1073437989c7","Type":"ContainerStarted","Data":"8cd7725bda9a350f1e07852ec4462eacd1fc19b25d689b338594e9d712327cd4"}
Nov 01 03:51:29.589652 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:29.589601    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s" podStartSLOduration=1.9017491739999999 podStartE2EDuration="2.589587281s" totalImagesPullingTime="687.838107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:29.588690259 +0000 UTC m=+558.150723276" watchObservedRunningTime="2026-11-01 03:51:29.589587281 +0000 UTC m=+558.151620298"
Nov 01 03:51:30.140189 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:30.140142    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:35.194960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.194923    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:35.207077 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.207052    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:35.396184 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.396144    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tqv66"]
Nov 01 03:51:35.399899 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.399875    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.448736 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.448655    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 01 03:51:35.455536 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.455505    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tqv66"]
Nov 01 03:51:35.509612 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.509578    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.509802 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.509646    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tzld\" (UniqueName: \"kubernetes.io/projected/f26ff8f3-4414-4233-b7e3-343c78a767b7-kube-api-access-5tzld\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.509802 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.509718    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f26ff8f3-4414-4233-b7e3-343c78a767b7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.509802 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.509779    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-home\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.509912 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.509889    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.509950 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.509936    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.523487 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.523468    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tzkq2"
Nov 01 03:51:35.611210 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.611176    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-home\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.611379 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.611226    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.611379 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.611338    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.611525 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.611393    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.611525 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.611457    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzld\" (UniqueName: \"kubernetes.io/projected/f26ff8f3-4414-4233-b7e3-343c78a767b7-kube-api-access-5tzld\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.611525 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.611477    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f26ff8f3-4414-4233-b7e3-343c78a767b7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.611682 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.611592    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.611682 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.611629    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-home\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.611846 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.611828    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.613528 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.613507    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f26ff8f3-4414-4233-b7e3-343c78a767b7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.613881 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.613866    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f26ff8f3-4414-4233-b7e3-343c78a767b7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.673693 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.673668    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzld\" (UniqueName: \"kubernetes.io/projected/f26ff8f3-4414-4233-b7e3-343c78a767b7-kube-api-access-5tzld\") pod \"echo-simulator-kserve-64b6f8f96c-tqv66\" (UID: \"f26ff8f3-4414-4233-b7e3-343c78a767b7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:35.712204 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:35.712122    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:36.089823 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:36.089735    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tqv66"]
Nov 01 03:51:36.097670 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:51:36.097643    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26ff8f3_4414_4233_b7e3_343c78a767b7.slice/crio-c5fc9119058f4eb740d40c2435fa3559c0563f9a514ad57cfa3ab089eaa129a8 WatchSource:0}: Error finding container c5fc9119058f4eb740d40c2435fa3559c0563f9a514ad57cfa3ab089eaa129a8: Status 404 returned error can't find the container with id c5fc9119058f4eb740d40c2435fa3559c0563f9a514ad57cfa3ab089eaa129a8
Nov 01 03:51:36.518673 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:36.518633    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66" event={"ID":"f26ff8f3-4414-4233-b7e3-343c78a767b7","Type":"ContainerStarted","Data":"c5fc9119058f4eb740d40c2435fa3559c0563f9a514ad57cfa3ab089eaa129a8"}
Nov 01 03:51:38.140906 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:38.140869    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:38.162187 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:38.162152    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:38.544624 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:38.544349    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6756s"
Nov 01 03:51:39.537651 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:39.537618    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66" event={"ID":"f26ff8f3-4414-4233-b7e3-343c78a767b7","Type":"ContainerStarted","Data":"4536ed1947f0182ad0b04dcd45246d225d8ca95d587f584a05b55f6f985240fb"}
Nov 01 03:51:39.654954 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:39.654900    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66" podStartSLOduration=1.687734662 podStartE2EDuration="4.654885315s" totalImagesPullingTime="2.967150653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:39.65282518 +0000 UTC m=+568.214858212" watchObservedRunningTime="2026-11-01 03:51:39.654885315 +0000 UTC m=+568.216918334"
Nov 01 03:51:39.713128 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:39.713097    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:42.039331 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.039290    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"]
Nov 01 03:51:42.043329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.043306    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.137738 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.137703    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 01 03:51:42.139997 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.139968    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"]
Nov 01 03:51:42.179370 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.179335    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhhvs\" (UniqueName: \"kubernetes.io/projected/91fcd874-e74a-4bf5-a6ca-3842296d22b7-kube-api-access-dhhvs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.179560 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.179437    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.179560 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.179522    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.179672 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.179562    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91fcd874-e74a-4bf5-a6ca-3842296d22b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.179672 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.179588    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.179672 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.179638    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.281083 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.281049    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhhvs\" (UniqueName: \"kubernetes.io/projected/91fcd874-e74a-4bf5-a6ca-3842296d22b7-kube-api-access-dhhvs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.281272 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.281091    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.281272 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.281111    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.281272 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.281128    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91fcd874-e74a-4bf5-a6ca-3842296d22b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.281272 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.281190    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.281272 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.281232    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.281634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.281612    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.281698 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.281645    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.281698 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.281678    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.283517 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.283499    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91fcd874-e74a-4bf5-a6ca-3842296d22b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.283592 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.283500    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91fcd874-e74a-4bf5-a6ca-3842296d22b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.353837 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.353771    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhhvs\" (UniqueName: \"kubernetes.io/projected/91fcd874-e74a-4bf5-a6ca-3842296d22b7-kube-api-access-dhhvs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52\" (UID: \"91fcd874-e74a-4bf5-a6ca-3842296d22b7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.355570 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.355548    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:42.738543 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:42.738505    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"]
Nov 01 03:51:42.746070 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:51:42.746045    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91fcd874_e74a_4bf5_a6ca_3842296d22b7.slice/crio-7a06337cc3884c5a3be6aa570b6c5c351e68c8bfb77ad9ad0591620be59fcd3d WatchSource:0}: Error finding container 7a06337cc3884c5a3be6aa570b6c5c351e68c8bfb77ad9ad0591620be59fcd3d: Status 404 returned error can't find the container with id 7a06337cc3884c5a3be6aa570b6c5c351e68c8bfb77ad9ad0591620be59fcd3d
Nov 01 03:51:43.557688 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:43.557593    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52" event={"ID":"91fcd874-e74a-4bf5-a6ca-3842296d22b7","Type":"ContainerStarted","Data":"8d4e2ce5e41eff80f36a52be219ad54326de40042a7773aa84ddb6ef39d8f421"}
Nov 01 03:51:43.557688 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:43.557639    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52" event={"ID":"91fcd874-e74a-4bf5-a6ca-3842296d22b7","Type":"ContainerStarted","Data":"7a06337cc3884c5a3be6aa570b6c5c351e68c8bfb77ad9ad0591620be59fcd3d"}
Nov 01 03:51:43.668272 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:43.668222    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52" podStartSLOduration=2.108802443 podStartE2EDuration="2.668209305s" totalImagesPullingTime="559.406862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:43.666569857 +0000 UTC m=+572.228602875" watchObservedRunningTime="2026-11-01 03:51:43.668209305 +0000 UTC m=+572.230242322"
Nov 01 03:51:44.355846 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:44.355815    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:45.713053 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:45.713012    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:45.726150 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:45.726122    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:46.584846 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:46.584813    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tqv66"
Nov 01 03:51:52.081355 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.081270    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"]
Nov 01 03:51:52.085264 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.085244    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.182718 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.182672    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"]
Nov 01 03:51:52.280004 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.279967    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.280175 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.280049    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.280175 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.280116    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.280246 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.280198    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v86t\" (UniqueName: \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-kube-api-access-9v86t\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.280286 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.280247    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.280319 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.280291    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.280319 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.280313    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.280380 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.280336    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.280416 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.280391    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.355968 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.355885    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:52.368746 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.368721    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:52.381275 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381250    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.381412 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381299    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v86t\" (UniqueName: \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-kube-api-access-9v86t\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.381412 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381387    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.381557 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381421    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.381557 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381470    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.381557 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381507    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.381557 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381545    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.381774 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381595    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.381774 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381623    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.381774 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381716    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.381889 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381865    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.381960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.381942    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.382090 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.382064    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.382209 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.382191    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.384174 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.384149    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.384282 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.384265    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.439166 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.439134    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.439277 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.439252    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v86t\" (UniqueName: \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-kube-api-access-9v86t\") pod \"maas-default-gateway-openshift-default-5df48fcc99-f6565\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:52.613251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.613172    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 03:51:52.697859 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:52.697828    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:53.129997 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:53.129962    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"]
Nov 01 03:51:53.138135 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:51:53.138104    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821ecb15_ab45_430e_8fc7_b7b1d9b069fd.slice/crio-c505e7452399efcd5a3928cfedf398ebf93525beba102ff693495a6aee559f08 WatchSource:0}: Error finding container c505e7452399efcd5a3928cfedf398ebf93525beba102ff693495a6aee559f08: Status 404 returned error can't find the container with id c505e7452399efcd5a3928cfedf398ebf93525beba102ff693495a6aee559f08
Nov 01 03:51:53.140405 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:53.140373    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:51:53.140518 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:53.140451    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:51:53.140518 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:53.140480    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:51:53.622164 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:53.622111    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" event={"ID":"821ecb15-ab45-430e-8fc7-b7b1d9b069fd","Type":"ContainerStarted","Data":"7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0"}
Nov 01 03:51:53.622363 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:53.622191    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" event={"ID":"821ecb15-ab45-430e-8fc7-b7b1d9b069fd","Type":"ContainerStarted","Data":"c505e7452399efcd5a3928cfedf398ebf93525beba102ff693495a6aee559f08"}
Nov 01 03:51:53.698644 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:53.698613    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:51:53.739391 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:53.739335    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" podStartSLOduration=1.73931914 podStartE2EDuration="1.73931914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:53.738527795 +0000 UTC m=+582.300560812" watchObservedRunningTime="2026-11-01 03:51:53.73931914 +0000 UTC m=+582.301352160"
Nov 01 03:51:54.699501 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:54.699468    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-f6565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 03:51:54.699975 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:54.699529    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" podUID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 03:51:55.698789 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:55.698758    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-f6565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 03:51:55.698963 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:55.698816    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" podUID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 03:51:56.698438 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:56.698403    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-f6565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 03:51:56.698835 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:56.698474    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" podUID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 03:51:57.698301 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:57.698266    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-f6565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 03:51:57.698492 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:57.698316    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" podUID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 03:51:58.698867 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:58.698822    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-f6565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 03:51:58.699346 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:58.698895    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" podUID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 03:51:58.722310 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:51:58.722276    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:51:59.698725 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:59.698691    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-f6565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 03:51:59.698911 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:51:59.698773    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" podUID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 03:52:01.377033 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:01.376978    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:52:01.377483 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:01.377363    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:52:01.378171 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:01.378148    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 03:52:10.725279 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:10.725241    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f87c44d4-d554c"]
Nov 01 03:52:10.730708 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:10.730689    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f87c44d4-d554c"
Nov 01 03:52:10.783028 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:10.783002    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f87c44d4-d554c"]
Nov 01 03:52:10.855350 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:10.855312    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwxz\" (UniqueName: \"kubernetes.io/projected/ecee51df-9d85-4121-bfa9-473fa0d54088-kube-api-access-skwxz\") pod \"authorino-5f87c44d4-d554c\" (UID: \"ecee51df-9d85-4121-bfa9-473fa0d54088\") " pod="kuadrant-system/authorino-5f87c44d4-d554c"
Nov 01 03:52:10.855550 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:10.855473    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ecee51df-9d85-4121-bfa9-473fa0d54088-tls-cert\") pod \"authorino-5f87c44d4-d554c\" (UID: \"ecee51df-9d85-4121-bfa9-473fa0d54088\") " pod="kuadrant-system/authorino-5f87c44d4-d554c"
Nov 01 03:52:10.956737 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:10.956706    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwxz\" (UniqueName: \"kubernetes.io/projected/ecee51df-9d85-4121-bfa9-473fa0d54088-kube-api-access-skwxz\") pod \"authorino-5f87c44d4-d554c\" (UID: \"ecee51df-9d85-4121-bfa9-473fa0d54088\") " pod="kuadrant-system/authorino-5f87c44d4-d554c"
Nov 01 03:52:10.956889 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:10.956768    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ecee51df-9d85-4121-bfa9-473fa0d54088-tls-cert\") pod \"authorino-5f87c44d4-d554c\" (UID: \"ecee51df-9d85-4121-bfa9-473fa0d54088\") " pod="kuadrant-system/authorino-5f87c44d4-d554c"
Nov 01 03:52:10.959142 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:10.959123    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ecee51df-9d85-4121-bfa9-473fa0d54088-tls-cert\") pod \"authorino-5f87c44d4-d554c\" (UID: \"ecee51df-9d85-4121-bfa9-473fa0d54088\") " pod="kuadrant-system/authorino-5f87c44d4-d554c"
Nov 01 03:52:11.013720 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:11.013658    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwxz\" (UniqueName: \"kubernetes.io/projected/ecee51df-9d85-4121-bfa9-473fa0d54088-kube-api-access-skwxz\") pod \"authorino-5f87c44d4-d554c\" (UID: \"ecee51df-9d85-4121-bfa9-473fa0d54088\") " pod="kuadrant-system/authorino-5f87c44d4-d554c"
Nov 01 03:52:11.042791 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:11.042765    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f87c44d4-d554c"
Nov 01 03:52:11.412808 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:11.412730    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f87c44d4-d554c"]
Nov 01 03:52:11.420911 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:52:11.420883    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecee51df_9d85_4121_bfa9_473fa0d54088.slice/crio-7109500a21d4dd3eeba6936f57f13b42d22e9b5608aa0b72d1c60856242d483a WatchSource:0}: Error finding container 7109500a21d4dd3eeba6936f57f13b42d22e9b5608aa0b72d1c60856242d483a: Status 404 returned error can't find the container with id 7109500a21d4dd3eeba6936f57f13b42d22e9b5608aa0b72d1c60856242d483a
Nov 01 03:52:11.652472 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:11.652446    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:52:11.653358 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:11.653332    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:52:11.716043 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:11.716012    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f87c44d4-d554c" event={"ID":"ecee51df-9d85-4121-bfa9-473fa0d54088","Type":"ContainerStarted","Data":"7109500a21d4dd3eeba6936f57f13b42d22e9b5608aa0b72d1c60856242d483a"}
Nov 01 03:52:13.726963 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:13.726925    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f87c44d4-d554c" event={"ID":"ecee51df-9d85-4121-bfa9-473fa0d54088","Type":"ContainerStarted","Data":"e32d5b7871aa129509e52d93fca49e109f55e74f692d3eb7d79d42459a7de5d9"}
Nov 01 03:52:13.836151 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:13.836088    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f87c44d4-d554c" podStartSLOduration=2.052727416 podStartE2EDuration="3.836074551s" totalImagesPullingTime="1.783347135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:13.835885792 +0000 UTC m=+602.397918810" watchObservedRunningTime="2026-11-01 03:52:13.836074551 +0000 UTC m=+602.398107566"
Nov 01 03:52:13.884199 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:13.884163    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-664b5b9867-x4lkz"]
Nov 01 03:52:13.884405 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:13.884376    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-664b5b9867-x4lkz" podUID="046a104e-7cbc-4530-99d9-7cab7717f09d" containerName="authorino" containerID="cri-o://276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d" gracePeriod=30
Nov 01 03:52:14.146178 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.146147    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-664b5b9867-x4lkz"
Nov 01 03:52:14.288781 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.288696    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046a104e-7cbc-4530-99d9-7cab7717f09d-kube-api-access-ld2rb\" (UniqueName: \"kubernetes.io/projected/046a104e-7cbc-4530-99d9-7cab7717f09d-kube-api-access-ld2rb\") pod \"046a104e-7cbc-4530-99d9-7cab7717f09d\" (UID: \"046a104e-7cbc-4530-99d9-7cab7717f09d\") "
Nov 01 03:52:14.288947 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.288821    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/046a104e-7cbc-4530-99d9-7cab7717f09d-tls-cert\" (UniqueName: \"kubernetes.io/secret/046a104e-7cbc-4530-99d9-7cab7717f09d-tls-cert\") pod \"046a104e-7cbc-4530-99d9-7cab7717f09d\" (UID: \"046a104e-7cbc-4530-99d9-7cab7717f09d\") "
Nov 01 03:52:14.290907 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.290870    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046a104e-7cbc-4530-99d9-7cab7717f09d-kube-api-access-ld2rb" pod "046a104e-7cbc-4530-99d9-7cab7717f09d" (UID: "046a104e-7cbc-4530-99d9-7cab7717f09d"). InnerVolumeSpecName "kube-api-access-ld2rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:14.302023 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.301994    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046a104e-7cbc-4530-99d9-7cab7717f09d-tls-cert" pod "046a104e-7cbc-4530-99d9-7cab7717f09d" (UID: "046a104e-7cbc-4530-99d9-7cab7717f09d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:14.389876 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.389840    4882 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/046a104e-7cbc-4530-99d9-7cab7717f09d-tls-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:52:14.389876 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.389880    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld2rb\" (UniqueName: \"kubernetes.io/projected/046a104e-7cbc-4530-99d9-7cab7717f09d-kube-api-access-ld2rb\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:52:14.735007 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.734964    4882 generic.go:356] "Generic (PLEG): container finished" podID="046a104e-7cbc-4530-99d9-7cab7717f09d" containerID="276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d" exitCode=0
Nov 01 03:52:14.736056 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.735021    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-664b5b9867-x4lkz"
Nov 01 03:52:14.736056 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.735096    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-664b5b9867-x4lkz" event={"ID":"046a104e-7cbc-4530-99d9-7cab7717f09d","Type":"ContainerDied","Data":"276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d"}
Nov 01 03:52:14.736056 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.735134    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-664b5b9867-x4lkz" event={"ID":"046a104e-7cbc-4530-99d9-7cab7717f09d","Type":"ContainerDied","Data":"b32387db1a2b1e64d6efdfd78d9a3d517301d608550b4d3fad6feb7cea9a9603"}
Nov 01 03:52:14.736056 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.735155    4882 scope.go:122] "RemoveContainer" containerID="276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d"
Nov 01 03:52:14.747175 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.747156    4882 scope.go:122] "RemoveContainer" containerID="276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d"
Nov 01 03:52:14.747525 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:52:14.747444    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d\": container with ID starting with 276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d not found: ID does not exist" containerID="276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d"
Nov 01 03:52:14.747525 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.747477    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d"} err="failed to get container status \"276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d\": rpc error: code = NotFound desc = could not find container \"276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d\": container with ID starting with 276180c14bb940bee20d73eb5193ca82365e6ea2907de5800920bb35e8e52c0d not found: ID does not exist"
Nov 01 03:52:14.887863 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.887833    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-664b5b9867-x4lkz"]
Nov 01 03:52:14.896045 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:14.896008    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-664b5b9867-x4lkz"]
Nov 01 03:52:15.600745 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:15.600700    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="046a104e-7cbc-4530-99d9-7cab7717f09d" path="/var/lib/kubelet/pods/046a104e-7cbc-4530-99d9-7cab7717f09d/volumes"
Nov 01 03:52:28.722612 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:52:28.722576    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:52:41.761321 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.761290    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"]
Nov 01 03:52:41.761843 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.761756    4882 cpu_manager.go:414] "containerMap: removing container" podUID="046a104e-7cbc-4530-99d9-7cab7717f09d" containerName="authorino"
Nov 01 03:52:41.761843 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.761769    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046a104e-7cbc-4530-99d9-7cab7717f09d" containerName="authorino"
Nov 01 03:52:41.761918 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.761852    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="046a104e-7cbc-4530-99d9-7cab7717f09d" containerName="authorino"
Nov 01 03:52:41.765324 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.765304    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.814785 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.814744    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-48d2fd87-openshift-default-dockercfg-kffqq\""
Nov 01 03:52:41.865649 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.865616    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"]
Nov 01 03:52:41.872917 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.872891    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-data\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.873074 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.872925    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-credential-socket\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.873074 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.872951    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-socket\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.873185 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.873087    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istiod-ca-cert\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.873185 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.873177    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-podinfo\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.873265 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.873222    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-token\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.873265 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.873256    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-envoy\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.873351 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.873290    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-certs\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.873419 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.873401    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbgrb\" (UniqueName: \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-kube-api-access-pbgrb\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.974932 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.974891    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbgrb\" (UniqueName: \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-kube-api-access-pbgrb\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975131 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.974953    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-data\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975131 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.974982    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-credential-socket\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975131 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.975013    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-socket\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975306 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.975126    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istiod-ca-cert\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975306 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.975215    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-podinfo\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975306 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.975289    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-token\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.975334    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-envoy\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.975386    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-certs\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.975474    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-socket\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975643 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.975530    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-data\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975744 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.975722    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-credential-socket\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975808 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.975763    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-certs\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.975988 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.975967    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istiod-ca-cert\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.977630 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.977610    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-envoy\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:41.977855 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:41.977837    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-podinfo\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:42.032219 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.032160    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-token\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:42.032412 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.032391    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbgrb\" (UniqueName: \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-kube-api-access-pbgrb\") pod \"e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") " pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:42.078236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.078210    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:42.379735 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.379637    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"]
Nov 01 03:52:42.388035 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.388009    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.447171 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.447129    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"]
Nov 01 03:52:42.448167 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.448143    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f50c3be9-gw-openshift-default-dockercfg-k4wdt\""
Nov 01 03:52:42.481283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.481238    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff0ff2b-a199-46db-a090-b563f32e38b0-istiod-ca-cert\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.481451 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.481343    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-data\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.481451 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.481393    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-socket\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.481451 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.481411    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc7lk\" (UniqueName: \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-kube-api-access-gc7lk\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.481559 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.481451    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-certs\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.481608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.481587    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-podinfo\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.481654 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.481643    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-credential-socket\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.481723 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.481701    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-envoy\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.481776 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.481759    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-token\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.551055 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.551019    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"]
Nov 01 03:52:42.559306 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:52:42.559275    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45c023d_f0c7_4350_9e31_29eabeaa6f11.slice/crio-625cc2d8f09f0d2f4e1ab9d86f34e3f2321ac7c99fe1c12a8edc66bf577bcff8 WatchSource:0}: Error finding container 625cc2d8f09f0d2f4e1ab9d86f34e3f2321ac7c99fe1c12a8edc66bf577bcff8: Status 404 returned error can't find the container with id 625cc2d8f09f0d2f4e1ab9d86f34e3f2321ac7c99fe1c12a8edc66bf577bcff8
Nov 01 03:52:42.561387 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.561354    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:52:42.561944 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.561418    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:52:42.561944 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.561473    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:52:42.582654 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.582628    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-socket\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.582751 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.582662    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc7lk\" (UniqueName: \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-kube-api-access-gc7lk\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.582751 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.582681    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-certs\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.582751 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.582715    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-podinfo\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.582909 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.582759    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-credential-socket\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.582909 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.582775    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-envoy\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.582909 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.582790    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-token\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.582909 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.582863    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff0ff2b-a199-46db-a090-b563f32e38b0-istiod-ca-cert\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.582909 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.582904    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-data\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.583154 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.583096    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-socket\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.583154 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.583130    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-credential-socket\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.583250 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.583185    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-certs\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.583286 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.583242    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-data\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.583822 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.583793    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff0ff2b-a199-46db-a090-b563f32e38b0-istiod-ca-cert\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.585152 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.585129    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-envoy\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.585241 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.585227    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-podinfo\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.641597 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.641524    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc7lk\" (UniqueName: \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-kube-api-access-gc7lk\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.643096 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.643068    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-token\") pod \"e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") " pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.700828 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.700786    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:42.898368 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.898280    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" event={"ID":"e45c023d-f0c7-4350-9e31-29eabeaa6f11","Type":"ContainerStarted","Data":"d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528"}
Nov 01 03:52:42.898368 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:42.898339    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" event={"ID":"e45c023d-f0c7-4350-9e31-29eabeaa6f11","Type":"ContainerStarted","Data":"625cc2d8f09f0d2f4e1ab9d86f34e3f2321ac7c99fe1c12a8edc66bf577bcff8"}
Nov 01 03:52:43.015197 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:43.015143    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" podStartSLOduration=2.015129309 podStartE2EDuration="2.015129309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:43.013096053 +0000 UTC m=+631.575129105" watchObservedRunningTime="2026-11-01 03:52:43.015129309 +0000 UTC m=+631.577162325"
Nov 01 03:52:43.078998 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:43.078963    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:43.084003 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:43.083980    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:43.129834 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:43.129805    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"]
Nov 01 03:52:43.137633 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:52:43.137607    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff0ff2b_a199_46db_a090_b563f32e38b0.slice/crio-3106493d922ac4cda8359e76be9768ff1f31e5c988130f84e7750a9577ced243 WatchSource:0}: Error finding container 3106493d922ac4cda8359e76be9768ff1f31e5c988130f84e7750a9577ced243: Status 404 returned error can't find the container with id 3106493d922ac4cda8359e76be9768ff1f31e5c988130f84e7750a9577ced243
Nov 01 03:52:43.140072 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:43.140044    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:52:43.140174 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:43.140107    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:52:43.140174 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:43.140137    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:52:43.904719 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:43.904679    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" event={"ID":"7ff0ff2b-a199-46db-a090-b563f32e38b0","Type":"ContainerStarted","Data":"f561a44c87ec7931b7f39406b95dcf99162fabb538838a4f70521c3123890b79"}
Nov 01 03:52:43.905141 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:43.904740    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" event={"ID":"7ff0ff2b-a199-46db-a090-b563f32e38b0","Type":"ContainerStarted","Data":"3106493d922ac4cda8359e76be9768ff1f31e5c988130f84e7750a9577ced243"}
Nov 01 03:52:43.905141 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:43.904766    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:43.905866 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:43.905841    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:52:44.022142 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:44.022084    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" podStartSLOduration=2.022066789 podStartE2EDuration="2.022066789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:44.021601392 +0000 UTC m=+632.583634409" watchObservedRunningTime="2026-11-01 03:52:44.022066789 +0000 UTC m=+632.584099808"
Nov 01 03:52:44.701896 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:44.701861    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:44.707195 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:44.707168    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:44.910569 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:44.910540    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:44.911391 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:44.911374    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:52:48.957508 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:48.957475    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"]
Nov 01 03:52:48.962292 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:48.962272    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.012098 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.012068    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6800c12d-openshift-default-dockercfg-h8jgn\""
Nov 01 03:52:49.048023 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.047995    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-socket\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.048184 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.048032    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-data\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.048184 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.048127    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvbq\" (UniqueName: \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-kube-api-access-grvbq\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.048310 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.048182    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-certs\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.048310 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.048278    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-envoy\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.048413 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.048308    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istiod-ca-cert\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.048413 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.048338    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-podinfo\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.048543 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.048417    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-credential-socket\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.048543 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.048484    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-token\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.061632 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.061605    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"]
Nov 01 03:52:49.149876 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.149842    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-token\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150061 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.149888    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-socket\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150061 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150011    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-data\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150180 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvbq\" (UniqueName: \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-kube-api-access-grvbq\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150180 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150133    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-certs\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150331 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150200    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-envoy\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150331 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150232    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-socket\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150331 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150242    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istiod-ca-cert\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150331 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150292    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-podinfo\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150580 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150348    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-credential-socket\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150580 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150376    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-data\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150580 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150516    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-certs\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150711 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150689    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-credential-socket\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.150883 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.150856    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istiod-ca-cert\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.152412 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.152393    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-envoy\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.152723 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.152705    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-podinfo\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.208237 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.208159    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-token\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.209527 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.209497    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvbq\" (UniqueName: \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-kube-api-access-grvbq\") pod \"e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") " pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.275229 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.275202    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:49.702872 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.702835    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"]
Nov 01 03:52:49.710768 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:52:49.710743    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7ad2d2_f9d6_49fc_b80f_2e6ad8ae85de.slice/crio-2ff3a312b9203bc88c5b4287447af5013edd0bc8ec6c41828718821fbdcf9aa3 WatchSource:0}: Error finding container 2ff3a312b9203bc88c5b4287447af5013edd0bc8ec6c41828718821fbdcf9aa3: Status 404 returned error can't find the container with id 2ff3a312b9203bc88c5b4287447af5013edd0bc8ec6c41828718821fbdcf9aa3
Nov 01 03:52:49.712946 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.712916    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:52:49.713041 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.712981    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:52:49.713041 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.713006    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:52:49.936365 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.936329    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" event={"ID":"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de","Type":"ContainerStarted","Data":"b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff"}
Nov 01 03:52:49.936573 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:49.936388    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" event={"ID":"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de","Type":"ContainerStarted","Data":"2ff3a312b9203bc88c5b4287447af5013edd0bc8ec6c41828718821fbdcf9aa3"}
Nov 01 03:52:50.049958 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:50.049852    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" podStartSLOduration=2.049838419 podStartE2EDuration="2.049838419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:50.049332938 +0000 UTC m=+638.611365956" watchObservedRunningTime="2026-11-01 03:52:50.049838419 +0000 UTC m=+638.611871435"
Nov 01 03:52:50.275932 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:50.275890    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:50.280949 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:50.280925    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:50.942133 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:50.942098    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:50.942989 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:50.942964    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:52:56.288962 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.288926    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"]
Nov 01 03:52:56.292727 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.292704    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:52:56.341419 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.341389    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4c476fff-dockercfg-xq4l7\""
Nov 01 03:52:56.387262 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.387227    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4c476fff\""
Nov 01 03:52:56.389543 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.389521    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"]
Nov 01 03:52:56.424892 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.424854    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq\" (UID: \"733da0a6-4de6-4e23-b2be-bc85b00fe9c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:52:56.425066 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.424924    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcnkx\" (UniqueName: \"kubernetes.io/projected/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-kube-api-access-tcnkx\") pod \"payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq\" (UID: \"733da0a6-4de6-4e23-b2be-bc85b00fe9c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:52:56.525607 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.525572    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq\" (UID: \"733da0a6-4de6-4e23-b2be-bc85b00fe9c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:52:56.525779 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.525711    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcnkx\" (UniqueName: \"kubernetes.io/projected/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-kube-api-access-tcnkx\") pod \"payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq\" (UID: \"733da0a6-4de6-4e23-b2be-bc85b00fe9c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:52:56.526351 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.526333    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq\" (UID: \"733da0a6-4de6-4e23-b2be-bc85b00fe9c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:52:56.583711 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.583639    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcnkx\" (UniqueName: \"kubernetes.io/projected/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-kube-api-access-tcnkx\") pod \"payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq\" (UID: \"733da0a6-4de6-4e23-b2be-bc85b00fe9c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:52:56.606363 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.606328    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:52:56.983343 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.983307    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"]
Nov 01 03:52:56.991139 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:52:56.991111    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733da0a6_4de6_4e23_b2be_bc85b00fe9c0.slice/crio-3271d4608bc15365be3786d08e17d76ef6d82408e925f54aa2c77ddcc0c18999 WatchSource:0}: Error finding container 3271d4608bc15365be3786d08e17d76ef6d82408e925f54aa2c77ddcc0c18999: Status 404 returned error can't find the container with id 3271d4608bc15365be3786d08e17d76ef6d82408e925f54aa2c77ddcc0c18999
Nov 01 03:52:56.993073 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:56.993057    4882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:52:57.984334 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:57.984294    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq" event={"ID":"733da0a6-4de6-4e23-b2be-bc85b00fe9c0","Type":"ContainerStarted","Data":"3271d4608bc15365be3786d08e17d76ef6d82408e925f54aa2c77ddcc0c18999"}
Nov 01 03:52:58.755212 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:52:58.755140    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:52:58.839642 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:58.839609    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"]
Nov 01 03:52:58.844879 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:58.844855    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:52:58.898590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:58.898558    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"]
Nov 01 03:52:58.940766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:58.940734    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3145c214\""
Nov 01 03:52:58.940766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:58.940765    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3145c214-dockercfg-gnklx\""
Nov 01 03:52:58.954576 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:58.954538    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302961d2-279f-4a93-abcd-264733f897b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb\" (UID: \"302961d2-279f-4a93-abcd-264733f897b2\") " pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:52:58.954784 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:58.954760    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtz2r\" (UniqueName: \"kubernetes.io/projected/302961d2-279f-4a93-abcd-264733f897b2-kube-api-access-jtz2r\") pod \"payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb\" (UID: \"302961d2-279f-4a93-abcd-264733f897b2\") " pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:52:59.055814 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:59.055725    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtz2r\" (UniqueName: \"kubernetes.io/projected/302961d2-279f-4a93-abcd-264733f897b2-kube-api-access-jtz2r\") pod \"payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb\" (UID: \"302961d2-279f-4a93-abcd-264733f897b2\") " pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:52:59.056324 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:59.055843    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302961d2-279f-4a93-abcd-264733f897b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb\" (UID: \"302961d2-279f-4a93-abcd-264733f897b2\") " pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:52:59.056788 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:59.056759    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302961d2-279f-4a93-abcd-264733f897b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb\" (UID: \"302961d2-279f-4a93-abcd-264733f897b2\") " pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:52:59.113127 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:59.113098    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtz2r\" (UniqueName: \"kubernetes.io/projected/302961d2-279f-4a93-abcd-264733f897b2-kube-api-access-jtz2r\") pod \"payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb\" (UID: \"302961d2-279f-4a93-abcd-264733f897b2\") " pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:52:59.165050 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:59.165019    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:52:59.553814 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:59.553779    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"]
Nov 01 03:52:59.560956 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:52:59.560929    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302961d2_279f_4a93_abcd_264733f897b2.slice/crio-0ecc393f9ca69c384cfbd25d8a1f280f7114d7c46ad3b8c9c876c169d95635ee WatchSource:0}: Error finding container 0ecc393f9ca69c384cfbd25d8a1f280f7114d7c46ad3b8c9c876c169d95635ee: Status 404 returned error can't find the container with id 0ecc393f9ca69c384cfbd25d8a1f280f7114d7c46ad3b8c9c876c169d95635ee
Nov 01 03:52:59.997366 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:59.997310    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq" event={"ID":"733da0a6-4de6-4e23-b2be-bc85b00fe9c0","Type":"ContainerStarted","Data":"3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1"}
Nov 01 03:52:59.997551 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:59.997386    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:52:59.998619 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:52:59.998592    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb" event={"ID":"302961d2-279f-4a93-abcd-264733f897b2","Type":"ContainerStarted","Data":"0ecc393f9ca69c384cfbd25d8a1f280f7114d7c46ad3b8c9c876c169d95635ee"}
Nov 01 03:53:00.114610 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:00.114508    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq" podStartSLOduration=1.352496864 podStartE2EDuration="4.114494453s" totalImagesPullingTime="2.761997589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:53:00.112465041 +0000 UTC m=+648.674498055" watchObservedRunningTime="2026-11-01 03:53:00.114494453 +0000 UTC m=+648.676527470"
Nov 01 03:53:01.005386 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:01.005338    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb" event={"ID":"302961d2-279f-4a93-abcd-264733f897b2","Type":"ContainerStarted","Data":"463ebe7dea7bd547d7557e330a6d4f7a9e1c3766728c2a42374b6cef0d4f02b9"}
Nov 01 03:53:01.005601 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:01.005541    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:53:01.115776 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:01.115718    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb" podStartSLOduration=2.253555899 podStartE2EDuration="3.115702746s" totalImagesPullingTime="862.146847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:53:01.114847974 +0000 UTC m=+649.676880987" watchObservedRunningTime="2026-11-01 03:53:01.115702746 +0000 UTC m=+649.677735742"
Nov 01 03:53:05.393339 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.393304    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"]
Nov 01 03:53:05.397127 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.397105    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:53:05.445771 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.445737    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6800c12d\""
Nov 01 03:53:05.498221 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.498191    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6800c12d-dockercfg-rt858\""
Nov 01 03:53:05.511025 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.510912    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"]
Nov 01 03:53:05.532518 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.532478    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxh25\" (UniqueName: \"kubernetes.io/projected/fcf77613-8732-4696-9cde-80e5da94e4e9-kube-api-access-kxh25\") pod \"payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls\" (UID: \"fcf77613-8732-4696-9cde-80e5da94e4e9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:53:05.532696 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.532557    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf77613-8732-4696-9cde-80e5da94e4e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls\" (UID: \"fcf77613-8732-4696-9cde-80e5da94e4e9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:53:05.559196 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.559163    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"]
Nov 01 03:53:05.563096 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.563076    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:53:05.617737 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.617702    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"]
Nov 01 03:53:05.636587 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.634815    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxh25\" (UniqueName: \"kubernetes.io/projected/fcf77613-8732-4696-9cde-80e5da94e4e9-kube-api-access-kxh25\") pod \"payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls\" (UID: \"fcf77613-8732-4696-9cde-80e5da94e4e9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:53:05.636587 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.634911    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf77613-8732-4696-9cde-80e5da94e4e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls\" (UID: \"fcf77613-8732-4696-9cde-80e5da94e4e9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:53:05.636587 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.636381    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf77613-8732-4696-9cde-80e5da94e4e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls\" (UID: \"fcf77613-8732-4696-9cde-80e5da94e4e9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:53:05.692768 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.692730    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxh25\" (UniqueName: \"kubernetes.io/projected/fcf77613-8732-4696-9cde-80e5da94e4e9-kube-api-access-kxh25\") pod \"payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls\" (UID: \"fcf77613-8732-4696-9cde-80e5da94e4e9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:53:05.709688 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.709658    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:53:05.735710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.735677    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwgdj\" (UniqueName: \"kubernetes.io/projected/48ace860-7548-4aac-a0bf-ee667bb36d1c-kube-api-access-dwgdj\") pod \"payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5\" (UID: \"48ace860-7548-4aac-a0bf-ee667bb36d1c\") " pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:53:05.735854 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.735737    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ace860-7548-4aac-a0bf-ee667bb36d1c-plugins-config-volume\") pod \"payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5\" (UID: \"48ace860-7548-4aac-a0bf-ee667bb36d1c\") " pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:53:05.837022 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.836981    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwgdj\" (UniqueName: \"kubernetes.io/projected/48ace860-7548-4aac-a0bf-ee667bb36d1c-kube-api-access-dwgdj\") pod \"payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5\" (UID: \"48ace860-7548-4aac-a0bf-ee667bb36d1c\") " pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:53:05.837211 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.837040    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ace860-7548-4aac-a0bf-ee667bb36d1c-plugins-config-volume\") pod \"payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5\" (UID: \"48ace860-7548-4aac-a0bf-ee667bb36d1c\") " pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:53:05.837909 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.837883    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ace860-7548-4aac-a0bf-ee667bb36d1c-plugins-config-volume\") pod \"payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5\" (UID: \"48ace860-7548-4aac-a0bf-ee667bb36d1c\") " pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:53:05.894585 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:05.894548    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwgdj\" (UniqueName: \"kubernetes.io/projected/48ace860-7548-4aac-a0bf-ee667bb36d1c-kube-api-access-dwgdj\") pod \"payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5\" (UID: \"48ace860-7548-4aac-a0bf-ee667bb36d1c\") " pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:53:06.082643 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:06.082560    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"]
Nov 01 03:53:06.090601 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:53:06.090571    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf77613_8732_4696_9cde_80e5da94e4e9.slice/crio-43eb6e41dd80798a6513e202c46e0422bfdb3ccb6944079361b06cc89c0223cb WatchSource:0}: Error finding container 43eb6e41dd80798a6513e202c46e0422bfdb3ccb6944079361b06cc89c0223cb: Status 404 returned error can't find the container with id 43eb6e41dd80798a6513e202c46e0422bfdb3ccb6944079361b06cc89c0223cb
Nov 01 03:53:06.176078 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:06.176048    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:53:06.551522 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:06.551488    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"]
Nov 01 03:53:06.559024 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:53:06.558993    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ace860_7548_4aac_a0bf_ee667bb36d1c.slice/crio-282b917ed7e6d88ef3ea57a523cdcd0b7038047e7d75d0c219b304a7266afc5a WatchSource:0}: Error finding container 282b917ed7e6d88ef3ea57a523cdcd0b7038047e7d75d0c219b304a7266afc5a: Status 404 returned error can't find the container with id 282b917ed7e6d88ef3ea57a523cdcd0b7038047e7d75d0c219b304a7266afc5a
Nov 01 03:53:07.042615 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:07.042578    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5" event={"ID":"48ace860-7548-4aac-a0bf-ee667bb36d1c","Type":"ContainerStarted","Data":"ef81ca08d3c7398b03b86187991c092557cdfad031990ad3989fb8d22d617ba6"}
Nov 01 03:53:07.042802 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:07.042637    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5" event={"ID":"48ace860-7548-4aac-a0bf-ee667bb36d1c","Type":"ContainerStarted","Data":"282b917ed7e6d88ef3ea57a523cdcd0b7038047e7d75d0c219b304a7266afc5a"}
Nov 01 03:53:07.042802 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:07.042667    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:53:07.044128 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:07.044105    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls" event={"ID":"fcf77613-8732-4696-9cde-80e5da94e4e9","Type":"ContainerStarted","Data":"af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646"}
Nov 01 03:53:07.044297 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:07.044149    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls" event={"ID":"fcf77613-8732-4696-9cde-80e5da94e4e9","Type":"ContainerStarted","Data":"43eb6e41dd80798a6513e202c46e0422bfdb3ccb6944079361b06cc89c0223cb"}
Nov 01 03:53:07.044297 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:07.044197    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:53:07.152501 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:07.152440    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5" podStartSLOduration=2.152411014 podStartE2EDuration="2.152411014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:53:07.151183952 +0000 UTC m=+655.713216971" watchObservedRunningTime="2026-11-01 03:53:07.152411014 +0000 UTC m=+655.714444031"
Nov 01 03:53:07.260497 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:07.260443    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls" podStartSLOduration=2.260410582 podStartE2EDuration="2.260410582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:53:07.258649503 +0000 UTC m=+655.820682519" watchObservedRunningTime="2026-11-01 03:53:07.260410582 +0000 UTC m=+655.822443611"
Nov 01 03:53:11.008888 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:11.008855    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:53:12.013986 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:12.013902    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:53:18.052849 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:18.052818    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:53:18.053322 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:18.053139    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:53:20.146526 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:20.146493    4882 generic.go:356] "Generic (PLEG): container finished" podID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerID="f561a44c87ec7931b7f39406b95dcf99162fabb538838a4f70521c3123890b79" exitCode=0
Nov 01 03:53:20.147047 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:20.146532    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" event={"ID":"7ff0ff2b-a199-46db-a090-b563f32e38b0","Type":"ContainerDied","Data":"f561a44c87ec7931b7f39406b95dcf99162fabb538838a4f70521c3123890b79"}
Nov 01 03:53:20.147416 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:20.147402    4882 scope.go:122] "RemoveContainer" containerID="f561a44c87ec7931b7f39406b95dcf99162fabb538838a4f70521c3123890b79"
Nov 01 03:53:20.148670 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:20.148646    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:53:20.148733 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:20.148711    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:53:20.148774 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:20.148755    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:53:21.153883 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:21.153847    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" event={"ID":"7ff0ff2b-a199-46db-a090-b563f32e38b0","Type":"ContainerStarted","Data":"f562fce218e566b61e5e939e968e1168812d634dd75b34fd379c9108e1ac03f6"}
Nov 01 03:53:21.704454 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:21.703827    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:53:21.704454 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:21.703963    4882 patch_prober.go:29] interesting pod/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 01 03:53:21.704454 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:21.704011    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 01 03:53:22.708635 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:22.708581    4882 patch_prober.go:29] interesting pod/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 01 03:53:22.709287 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:22.709232    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 01 03:53:23.701351 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:23.701320    4882 patch_prober.go:29] interesting pod/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 01 03:53:23.701537 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:23.701381    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 01 03:53:24.702237 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:24.702190    4882 patch_prober.go:29] interesting pod/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 01 03:53:24.702803 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:24.702259    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 01 03:53:25.702110 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:25.702065    4882 patch_prober.go:29] interesting pod/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 01 03:53:25.702323 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:25.702127    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 01 03:53:26.706417 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:26.706387    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:53:26.706893 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:26.706760    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:53:26.707556 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:26.707537    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:53:28.219206 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:28.219172    4882 generic.go:356] "Generic (PLEG): container finished" podID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerID="d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528" exitCode=0
Nov 01 03:53:28.219819 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:28.219260    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" event={"ID":"e45c023d-f0c7-4350-9e31-29eabeaa6f11","Type":"ContainerDied","Data":"d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528"}
Nov 01 03:53:28.219819 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:28.219759    4882 scope.go:122] "RemoveContainer" containerID="d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528"
Nov 01 03:53:28.221160 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:28.221127    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:53:28.221274 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:28.221218    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:53:28.221397 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:28.221270    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:53:28.722950 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:53:28.722912    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:53:28.904850 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:28.904816    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:53:29.226108 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:29.226063    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" event={"ID":"e45c023d-f0c7-4350-9e31-29eabeaa6f11","Type":"ContainerStarted","Data":"92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3"}
Nov 01 03:53:30.082879 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:30.082845    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:53:30.087618 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:30.087590    4882 patch_prober.go:29] interesting pod/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 01 03:53:30.087737 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:30.087644    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 01 03:53:31.086817 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:31.086779    4882 patch_prober.go:29] interesting pod/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 01 03:53:31.087219 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:31.086836    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 01 03:53:32.082403 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:32.082367    4882 patch_prober.go:29] interesting pod/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 01 03:53:32.082725 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:32.082696    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 01 03:53:33.025457 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.025407    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"]
Nov 01 03:53:33.029435 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.029403    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:33.083413 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.083378    4882 patch_prober.go:29] interesting pod/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 01 03:53:33.083691 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.083664    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 01 03:53:33.087449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.085819    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"]
Nov 01 03:53:33.127931 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.127900    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:53:33.128058 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.127901    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 03:53:33.189151 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.189116    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cnz7\" (UniqueName: \"kubernetes.io/projected/4c283a47-e114-4920-a1c0-5344f540ed8d-kube-api-access-7cnz7\") pod \"maas-api-revoke-keys-e2e-ait-4c476fff-ftssr\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:33.189343 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.189168    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c476fff-ftssr\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:33.189414 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.189341    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c476fff-ftssr\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:33.290158 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.290074    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cnz7\" (UniqueName: \"kubernetes.io/projected/4c283a47-e114-4920-a1c0-5344f540ed8d-kube-api-access-7cnz7\") pod \"maas-api-revoke-keys-e2e-ait-4c476fff-ftssr\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:33.290158 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.290128    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c476fff-ftssr\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:33.290358 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.290210    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c476fff-ftssr\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:33.290836 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.290812    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c476fff-ftssr\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:33.292646 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.292625    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c476fff-ftssr\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:33.347721 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.347685    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cnz7\" (UniqueName: \"kubernetes.io/projected/4c283a47-e114-4920-a1c0-5344f540ed8d-kube-api-access-7cnz7\") pod \"maas-api-revoke-keys-e2e-ait-4c476fff-ftssr\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:33.641764 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:33.641677    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:34.045083 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:34.045049    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"]
Nov 01 03:53:34.052517 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:53:34.052488    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c283a47_e114_4920_a1c0_5344f540ed8d.slice/crio-eb32f1b03ce901c6c91f0017876ed51ac4cd2d2e29d8d9c26d7d6096d2a99e79 WatchSource:0}: Error finding container eb32f1b03ce901c6c91f0017876ed51ac4cd2d2e29d8d9c26d7d6096d2a99e79: Status 404 returned error can't find the container with id eb32f1b03ce901c6c91f0017876ed51ac4cd2d2e29d8d9c26d7d6096d2a99e79
Nov 01 03:53:34.079237 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:34.079206    4882 patch_prober.go:29] interesting pod/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 01 03:53:34.079401 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:34.079257    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 01 03:53:34.254273 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:34.254232    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr" event={"ID":"4c283a47-e114-4920-a1c0-5344f540ed8d","Type":"ContainerStarted","Data":"eb32f1b03ce901c6c91f0017876ed51ac4cd2d2e29d8d9c26d7d6096d2a99e79"}
Nov 01 03:53:35.083039 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:35.083007    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:53:35.083469 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:35.083446    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:53:35.084172 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:35.084152    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:53:39.283482 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:39.283371    4882 generic.go:356] "Generic (PLEG): container finished" podID="4c283a47-e114-4920-a1c0-5344f540ed8d" containerID="c0675fc51cc965356375d9509d3299df94bce126885ce8dfd49b2b3295e9939f" exitCode=0
Nov 01 03:53:39.283482 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:39.283453    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr" event={"ID":"4c283a47-e114-4920-a1c0-5344f540ed8d","Type":"ContainerDied","Data":"c0675fc51cc965356375d9509d3299df94bce126885ce8dfd49b2b3295e9939f"}
Nov 01 03:53:39.790540 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:39.790509    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"]
Nov 01 03:53:39.794370 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:39.794347    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:39.849347 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:39.849317    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"]
Nov 01 03:53:39.961008 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:39.960974    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3145c214-lnml6\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:39.961180 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:39.961118    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3145c214-lnml6\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:39.961228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:39.961182    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmwj4\" (UniqueName: \"kubernetes.io/projected/aae2c650-ff58-4caf-ac45-0ca586275770-kube-api-access-wmwj4\") pod \"maas-api-revoke-keys-e2e-ait-3145c214-lnml6\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:40.062709 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.062628    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3145c214-lnml6\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:40.062709 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.062700    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3145c214-lnml6\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:40.062890 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.062742    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmwj4\" (UniqueName: \"kubernetes.io/projected/aae2c650-ff58-4caf-ac45-0ca586275770-kube-api-access-wmwj4\") pod \"maas-api-revoke-keys-e2e-ait-3145c214-lnml6\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:40.063273 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.063249    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3145c214-lnml6\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:40.065071 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.065051    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3145c214-lnml6\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:40.119814 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.119779    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmwj4\" (UniqueName: \"kubernetes.io/projected/aae2c650-ff58-4caf-ac45-0ca586275770-kube-api-access-wmwj4\") pod \"maas-api-revoke-keys-e2e-ait-3145c214-lnml6\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:40.407364 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.407338    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:40.447810 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.447777    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:40.568067 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.568036    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-openshift-ca\") pod \"4c283a47-e114-4920-a1c0-5344f540ed8d\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") "
Nov 01 03:53:40.568251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.568091    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c283a47-e114-4920-a1c0-5344f540ed8d-kube-api-access-7cnz7\" (UniqueName: \"kubernetes.io/projected/4c283a47-e114-4920-a1c0-5344f540ed8d-kube-api-access-7cnz7\") pod \"4c283a47-e114-4920-a1c0-5344f540ed8d\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") "
Nov 01 03:53:40.568251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.568189    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-xks-ca\") pod \"4c283a47-e114-4920-a1c0-5344f540ed8d\" (UID: \"4c283a47-e114-4920-a1c0-5344f540ed8d\") "
Nov 01 03:53:40.568489 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.568460    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-openshift-ca" pod "4c283a47-e114-4920-a1c0-5344f540ed8d" (UID: "4c283a47-e114-4920-a1c0-5344f540ed8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:40.570463 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.570412    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-xks-ca" pod "4c283a47-e114-4920-a1c0-5344f540ed8d" (UID: "4c283a47-e114-4920-a1c0-5344f540ed8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:40.570693 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.570672    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c283a47-e114-4920-a1c0-5344f540ed8d-kube-api-access-7cnz7" pod "4c283a47-e114-4920-a1c0-5344f540ed8d" (UID: "4c283a47-e114-4920-a1c0-5344f540ed8d"). InnerVolumeSpecName "kube-api-access-7cnz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:40.669450 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.669362    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:40.669450 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.669389    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cnz7\" (UniqueName: \"kubernetes.io/projected/4c283a47-e114-4920-a1c0-5344f540ed8d-kube-api-access-7cnz7\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:40.669450 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.669399    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c283a47-e114-4920-a1c0-5344f540ed8d-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:40.786568 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:40.786534    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"]
Nov 01 03:53:40.793642 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:53:40.793612    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae2c650_ff58_4caf_ac45_0ca586275770.slice/crio-98a12090a6f44ac4e9fbfb93bb7e2878e29485ea209cf3009db2db7012b39d8a WatchSource:0}: Error finding container 98a12090a6f44ac4e9fbfb93bb7e2878e29485ea209cf3009db2db7012b39d8a: Status 404 returned error can't find the container with id 98a12090a6f44ac4e9fbfb93bb7e2878e29485ea209cf3009db2db7012b39d8a
Nov 01 03:53:41.296050 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:41.295910    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6" event={"ID":"aae2c650-ff58-4caf-ac45-0ca586275770","Type":"ContainerStarted","Data":"0c1166e0cab500dfba3c3adfd86439e6f36de9f983ddfd01870c1d88f18e7ec6"}
Nov 01 03:53:41.296050 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:41.295966    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6" event={"ID":"aae2c650-ff58-4caf-ac45-0ca586275770","Type":"ContainerStarted","Data":"98a12090a6f44ac4e9fbfb93bb7e2878e29485ea209cf3009db2db7012b39d8a"}
Nov 01 03:53:41.297329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:41.297303    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr" event={"ID":"4c283a47-e114-4920-a1c0-5344f540ed8d","Type":"ContainerDied","Data":"eb32f1b03ce901c6c91f0017876ed51ac4cd2d2e29d8d9c26d7d6096d2a99e79"}
Nov 01 03:53:41.297485 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:41.297333    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"
Nov 01 03:53:41.297485 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:41.297341    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb32f1b03ce901c6c91f0017876ed51ac4cd2d2e29d8d9c26d7d6096d2a99e79"
Nov 01 03:53:41.409692 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:41.409642    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6" podStartSLOduration=2.409627273 podStartE2EDuration="2.409627273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:53:41.408817293 +0000 UTC m=+689.970850311" watchObservedRunningTime="2026-11-01 03:53:41.409627273 +0000 UTC m=+689.971660290"
Nov 01 03:53:42.304217 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.304185    4882 generic.go:356] "Generic (PLEG): container finished" podID="aae2c650-ff58-4caf-ac45-0ca586275770" containerID="0c1166e0cab500dfba3c3adfd86439e6f36de9f983ddfd01870c1d88f18e7ec6" exitCode=0
Nov 01 03:53:42.304422 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.304266    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6" event={"ID":"aae2c650-ff58-4caf-ac45-0ca586275770","Type":"ContainerDied","Data":"0c1166e0cab500dfba3c3adfd86439e6f36de9f983ddfd01870c1d88f18e7ec6"}
Nov 01 03:53:42.311503 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.311478    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"]
Nov 01 03:53:42.311948 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.311936    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4c283a47-e114-4920-a1c0-5344f540ed8d" containerName="revoke-keys"
Nov 01 03:53:42.311995 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.311949    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c283a47-e114-4920-a1c0-5344f540ed8d" containerName="revoke-keys"
Nov 01 03:53:42.312044 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.312036    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c283a47-e114-4920-a1c0-5344f540ed8d" containerName="revoke-keys"
Nov 01 03:53:42.315294 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.315273    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:42.369663 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.369462    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"]
Nov 01 03:53:42.489686 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.489653    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vhmj\" (UniqueName: \"kubernetes.io/projected/205ee3f8-b63a-4059-b40e-637d85219216-kube-api-access-7vhmj\") pod \"maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:42.490077 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.489693    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:42.490077 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.489878    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:42.590923 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.590844    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:42.591089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.590960    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:42.591089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.591008    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vhmj\" (UniqueName: \"kubernetes.io/projected/205ee3f8-b63a-4059-b40e-637d85219216-kube-api-access-7vhmj\") pod \"maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:42.591508 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.591485    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:42.593263 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.593239    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:42.648414 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.648389    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vhmj\" (UniqueName: \"kubernetes.io/projected/205ee3f8-b63a-4059-b40e-637d85219216-kube-api-access-7vhmj\") pod \"maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:42.928515 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:42.928424    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:43.302254 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:43.302218    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"]
Nov 01 03:53:43.309552 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:53:43.309515    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205ee3f8_b63a_4059_b40e_637d85219216.slice/crio-90fe433b2c362dce561d985f89f8af26bd9edf088c4dcfe0fb91df814ab0e7a5 WatchSource:0}: Error finding container 90fe433b2c362dce561d985f89f8af26bd9edf088c4dcfe0fb91df814ab0e7a5: Status 404 returned error can't find the container with id 90fe433b2c362dce561d985f89f8af26bd9edf088c4dcfe0fb91df814ab0e7a5
Nov 01 03:53:43.467291 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:43.467263    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:43.512353 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:43.512317    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-openshift-ca\") pod \"aae2c650-ff58-4caf-ac45-0ca586275770\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") "
Nov 01 03:53:43.512878 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:43.512385    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae2c650-ff58-4caf-ac45-0ca586275770-kube-api-access-wmwj4\" (UniqueName: \"kubernetes.io/projected/aae2c650-ff58-4caf-ac45-0ca586275770-kube-api-access-wmwj4\") pod \"aae2c650-ff58-4caf-ac45-0ca586275770\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") "
Nov 01 03:53:43.512878 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:43.512464    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-xks-ca\") pod \"aae2c650-ff58-4caf-ac45-0ca586275770\" (UID: \"aae2c650-ff58-4caf-ac45-0ca586275770\") "
Nov 01 03:53:43.512878 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:43.512764    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-openshift-ca" pod "aae2c650-ff58-4caf-ac45-0ca586275770" (UID: "aae2c650-ff58-4caf-ac45-0ca586275770"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:43.515017 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:43.514989    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-xks-ca" pod "aae2c650-ff58-4caf-ac45-0ca586275770" (UID: "aae2c650-ff58-4caf-ac45-0ca586275770"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:43.515128 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:43.515029    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae2c650-ff58-4caf-ac45-0ca586275770-kube-api-access-wmwj4" pod "aae2c650-ff58-4caf-ac45-0ca586275770" (UID: "aae2c650-ff58-4caf-ac45-0ca586275770"). InnerVolumeSpecName "kube-api-access-wmwj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:43.617268 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:43.615363    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:43.617268 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:43.615399    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmwj4\" (UniqueName: \"kubernetes.io/projected/aae2c650-ff58-4caf-ac45-0ca586275770-kube-api-access-wmwj4\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:43.617268 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:43.615414    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae2c650-ff58-4caf-ac45-0ca586275770-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:44.319069 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:44.319029    4882 generic.go:356] "Generic (PLEG): container finished" podID="205ee3f8-b63a-4059-b40e-637d85219216" containerID="325f228409c9c4976bc991d3603c89f69efb3ead349a494e3b88c07ebad32035" exitCode=0
Nov 01 03:53:44.319231 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:44.319125    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng" event={"ID":"205ee3f8-b63a-4059-b40e-637d85219216","Type":"ContainerDied","Data":"325f228409c9c4976bc991d3603c89f69efb3ead349a494e3b88c07ebad32035"}
Nov 01 03:53:44.319231 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:44.319197    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng" event={"ID":"205ee3f8-b63a-4059-b40e-637d85219216","Type":"ContainerStarted","Data":"90fe433b2c362dce561d985f89f8af26bd9edf088c4dcfe0fb91df814ab0e7a5"}
Nov 01 03:53:44.320563 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:44.320530    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6" event={"ID":"aae2c650-ff58-4caf-ac45-0ca586275770","Type":"ContainerDied","Data":"98a12090a6f44ac4e9fbfb93bb7e2878e29485ea209cf3009db2db7012b39d8a"}
Nov 01 03:53:44.320689 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:44.320571    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98a12090a6f44ac4e9fbfb93bb7e2878e29485ea209cf3009db2db7012b39d8a"
Nov 01 03:53:44.320689 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:44.320542    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"
Nov 01 03:53:45.465503 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:45.465479    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:45.532496 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:45.532461    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-xks-ca\") pod \"205ee3f8-b63a-4059-b40e-637d85219216\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") "
Nov 01 03:53:45.532643 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:45.532598    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-openshift-ca\") pod \"205ee3f8-b63a-4059-b40e-637d85219216\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") "
Nov 01 03:53:45.532706 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:45.532670    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205ee3f8-b63a-4059-b40e-637d85219216-kube-api-access-7vhmj\" (UniqueName: \"kubernetes.io/projected/205ee3f8-b63a-4059-b40e-637d85219216-kube-api-access-7vhmj\") pod \"205ee3f8-b63a-4059-b40e-637d85219216\" (UID: \"205ee3f8-b63a-4059-b40e-637d85219216\") "
Nov 01 03:53:45.532910 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:45.532885    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-openshift-ca" pod "205ee3f8-b63a-4059-b40e-637d85219216" (UID: "205ee3f8-b63a-4059-b40e-637d85219216"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:45.533035 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:45.533018    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:45.534642 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:45.534615    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-xks-ca" pod "205ee3f8-b63a-4059-b40e-637d85219216" (UID: "205ee3f8-b63a-4059-b40e-637d85219216"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:45.534838 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:45.534814    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205ee3f8-b63a-4059-b40e-637d85219216-kube-api-access-7vhmj" pod "205ee3f8-b63a-4059-b40e-637d85219216" (UID: "205ee3f8-b63a-4059-b40e-637d85219216"). InnerVolumeSpecName "kube-api-access-7vhmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:45.633741 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:45.633706    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vhmj\" (UniqueName: \"kubernetes.io/projected/205ee3f8-b63a-4059-b40e-637d85219216-kube-api-access-7vhmj\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:45.633880 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:45.633744    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/205ee3f8-b63a-4059-b40e-637d85219216-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:45.865656 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:45.865576    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"]
Nov 01 03:53:46.334325 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:46.334287    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng" event={"ID":"205ee3f8-b63a-4059-b40e-637d85219216","Type":"ContainerDied","Data":"90fe433b2c362dce561d985f89f8af26bd9edf088c4dcfe0fb91df814ab0e7a5"}
Nov 01 03:53:46.334536 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:46.334327    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"
Nov 01 03:53:46.334536 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:46.334338    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90fe433b2c362dce561d985f89f8af26bd9edf088c4dcfe0fb91df814ab0e7a5"
Nov 01 03:53:47.004037 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.003997    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"]
Nov 01 03:53:47.004728 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.004707    4882 cpu_manager.go:414] "containerMap: removing container" podUID="aae2c650-ff58-4caf-ac45-0ca586275770" containerName="revoke-keys"
Nov 01 03:53:47.004825 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.004729    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae2c650-ff58-4caf-ac45-0ca586275770" containerName="revoke-keys"
Nov 01 03:53:47.004825 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.004751    4882 cpu_manager.go:414] "containerMap: removing container" podUID="205ee3f8-b63a-4059-b40e-637d85219216" containerName="revoke-keys"
Nov 01 03:53:47.004825 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.004759    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205ee3f8-b63a-4059-b40e-637d85219216" containerName="revoke-keys"
Nov 01 03:53:47.004977 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.004901    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="205ee3f8-b63a-4059-b40e-637d85219216" containerName="revoke-keys"
Nov 01 03:53:47.004977 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.004925    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae2c650-ff58-4caf-ac45-0ca586275770" containerName="revoke-keys"
Nov 01 03:53:47.007958 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.007936    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:47.047574 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.047544    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:47.047574 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.047574    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:47.047774 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.047749    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nh6\" (UniqueName: \"kubernetes.io/projected/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-kube-api-access-s9nh6\") pod \"maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:47.059109 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.059081    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:53:47.059232 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.059091    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 03:53:47.062544 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.062519    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"]
Nov 01 03:53:47.150767 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.150716    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nh6\" (UniqueName: \"kubernetes.io/projected/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-kube-api-access-s9nh6\") pod \"maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:47.151001 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.150948    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:47.151001 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.150981    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:47.152044 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.152020    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:47.153949 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.153928    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:47.208018 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.207986    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nh6\" (UniqueName: \"kubernetes.io/projected/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-kube-api-access-s9nh6\") pod \"maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:47.321551 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.321466    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:47.700155 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:47.700117    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"]
Nov 01 03:53:47.706702 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:53:47.706676    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a9b728_6638_4a5f_8b86_99a3ed2ca971.slice/crio-cd2a0ca77deabc58a2391ef5d7d5703ffb0783c9cef66be97de5af829c0465dc WatchSource:0}: Error finding container cd2a0ca77deabc58a2391ef5d7d5703ffb0783c9cef66be97de5af829c0465dc: Status 404 returned error can't find the container with id cd2a0ca77deabc58a2391ef5d7d5703ffb0783c9cef66be97de5af829c0465dc
Nov 01 03:53:48.348649 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:48.348552    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb" event={"ID":"e8a9b728-6638-4a5f-8b86-99a3ed2ca971","Type":"ContainerStarted","Data":"4c207a2baa9c432695eec079f58bb27de3c935722de994a3bce772d38a10d2ef"}
Nov 01 03:53:48.348649 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:48.348606    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb" event={"ID":"e8a9b728-6638-4a5f-8b86-99a3ed2ca971","Type":"ContainerStarted","Data":"cd2a0ca77deabc58a2391ef5d7d5703ffb0783c9cef66be97de5af829c0465dc"}
Nov 01 03:53:48.457583 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:48.457533    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb" podStartSLOduration=2.457517731 podStartE2EDuration="2.457517731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:53:48.457211642 +0000 UTC m=+697.019244660" watchObservedRunningTime="2026-11-01 03:53:48.457517731 +0000 UTC m=+697.019550749"
Nov 01 03:53:49.355165 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:49.355132    4882 generic.go:356] "Generic (PLEG): container finished" podID="e8a9b728-6638-4a5f-8b86-99a3ed2ca971" containerID="4c207a2baa9c432695eec079f58bb27de3c935722de994a3bce772d38a10d2ef" exitCode=0
Nov 01 03:53:49.355602 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:49.355222    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb" event={"ID":"e8a9b728-6638-4a5f-8b86-99a3ed2ca971","Type":"ContainerDied","Data":"4c207a2baa9c432695eec079f58bb27de3c935722de994a3bce772d38a10d2ef"}
Nov 01 03:53:50.500586 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:50.500565    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:50.585054 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:50.585024    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-openshift-ca\") pod \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") "
Nov 01 03:53:50.585243 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:50.585061    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-xks-ca\") pod \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") "
Nov 01 03:53:50.585243 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:50.585216    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-kube-api-access-s9nh6\" (UniqueName: \"kubernetes.io/projected/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-kube-api-access-s9nh6\") pod \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\" (UID: \"e8a9b728-6638-4a5f-8b86-99a3ed2ca971\") "
Nov 01 03:53:50.585416 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:50.585393    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-openshift-ca" pod "e8a9b728-6638-4a5f-8b86-99a3ed2ca971" (UID: "e8a9b728-6638-4a5f-8b86-99a3ed2ca971"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:50.587062 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:50.587039    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-xks-ca" pod "e8a9b728-6638-4a5f-8b86-99a3ed2ca971" (UID: "e8a9b728-6638-4a5f-8b86-99a3ed2ca971"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:50.587150 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:50.587126    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-kube-api-access-s9nh6" pod "e8a9b728-6638-4a5f-8b86-99a3ed2ca971" (UID: "e8a9b728-6638-4a5f-8b86-99a3ed2ca971"). InnerVolumeSpecName "kube-api-access-s9nh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:50.686754 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:50.686720    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:50.686754 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:50.686751    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:50.686947 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:50.686766    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nh6\" (UniqueName: \"kubernetes.io/projected/e8a9b728-6638-4a5f-8b86-99a3ed2ca971-kube-api-access-s9nh6\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:50.866104 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:50.866054    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq" podUID="733da0a6-4de6-4e23-b2be-bc85b00fe9c0" containerName="payload-pre-processing" containerID="cri-o://3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1" gracePeriod=25
Nov 01 03:53:51.007282 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:53:51.007254    4882 logging.go:55] [core] [Channel #68 SubChannel #69]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.59:50052", ServerName: "10.132.0.59:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.59:50052: connect: connection refused"
Nov 01 03:53:51.098368 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.098337    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:53:51.191653 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.191626    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-kube-api-access-tcnkx\" (UniqueName: \"kubernetes.io/projected/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-kube-api-access-tcnkx\") pod \"733da0a6-4de6-4e23-b2be-bc85b00fe9c0\" (UID: \"733da0a6-4de6-4e23-b2be-bc85b00fe9c0\") "
Nov 01 03:53:51.191823 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.191691    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-plugins-config-volume\") pod \"733da0a6-4de6-4e23-b2be-bc85b00fe9c0\" (UID: \"733da0a6-4de6-4e23-b2be-bc85b00fe9c0\") "
Nov 01 03:53:51.192075 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.192052    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-plugins-config-volume" pod "733da0a6-4de6-4e23-b2be-bc85b00fe9c0" (UID: "733da0a6-4de6-4e23-b2be-bc85b00fe9c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:51.193946 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.193920    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-kube-api-access-tcnkx" pod "733da0a6-4de6-4e23-b2be-bc85b00fe9c0" (UID: "733da0a6-4de6-4e23-b2be-bc85b00fe9c0"). InnerVolumeSpecName "kube-api-access-tcnkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:51.265687 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.265613    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"]
Nov 01 03:53:51.292937 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.292906    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcnkx\" (UniqueName: \"kubernetes.io/projected/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-kube-api-access-tcnkx\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:51.292937 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.292932    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733da0a6-4de6-4e23-b2be-bc85b00fe9c0-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:51.369166 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.369139    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"
Nov 01 03:53:51.369354 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.369173    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb" event={"ID":"e8a9b728-6638-4a5f-8b86-99a3ed2ca971","Type":"ContainerDied","Data":"cd2a0ca77deabc58a2391ef5d7d5703ffb0783c9cef66be97de5af829c0465dc"}
Nov 01 03:53:51.369354 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.369215    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd2a0ca77deabc58a2391ef5d7d5703ffb0783c9cef66be97de5af829c0465dc"
Nov 01 03:53:51.370770 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.370743    4882 generic.go:356] "Generic (PLEG): container finished" podID="733da0a6-4de6-4e23-b2be-bc85b00fe9c0" containerID="3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1" exitCode=0
Nov 01 03:53:51.370882 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.370782    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq" event={"ID":"733da0a6-4de6-4e23-b2be-bc85b00fe9c0","Type":"ContainerDied","Data":"3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1"}
Nov 01 03:53:51.370882 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.370798    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"
Nov 01 03:53:51.370882 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.370838    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq" event={"ID":"733da0a6-4de6-4e23-b2be-bc85b00fe9c0","Type":"ContainerDied","Data":"3271d4608bc15365be3786d08e17d76ef6d82408e925f54aa2c77ddcc0c18999"}
Nov 01 03:53:51.370882 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.370867    4882 scope.go:122] "RemoveContainer" containerID="3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1"
Nov 01 03:53:51.381859 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.381543    4882 scope.go:122] "RemoveContainer" containerID="3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1"
Nov 01 03:53:51.381980 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:53:51.381865    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1\": container with ID starting with 3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1 not found: ID does not exist" containerID="3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1"
Nov 01 03:53:51.381980 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.381894    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1"} err="failed to get container status \"3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1\": rpc error: code = NotFound desc = could not find container \"3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1\": container with ID starting with 3f23330d9c1c1db9811a8ef95a2eb101f1372ceacd9330d107c94653a70463b1 not found: ID does not exist"
Nov 01 03:53:51.602400 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.602323    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"]
Nov 01 03:53:51.602400 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:51.602356    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq"]
Nov 01 03:53:52.006615 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:52.006573    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c476fff-78b48df47-stqpq" podUID="733da0a6-4de6-4e23-b2be-bc85b00fe9c0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.59:50052\" within 1s: context deadline exceeded"
Nov 01 03:53:53.599690 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:53.599653    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733da0a6-4de6-4e23-b2be-bc85b00fe9c0" path="/var/lib/kubelet/pods/733da0a6-4de6-4e23-b2be-bc85b00fe9c0/volumes"
Nov 01 03:53:56.280059 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.280016    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb" podUID="302961d2-279f-4a93-abcd-264733f897b2" containerName="payload-processing" containerID="cri-o://463ebe7dea7bd547d7557e330a6d4f7a9e1c3766728c2a42374b6cef0d4f02b9" gracePeriod=25
Nov 01 03:53:56.407637 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.407601    4882 generic.go:356] "Generic (PLEG): container finished" podID="302961d2-279f-4a93-abcd-264733f897b2" containerID="463ebe7dea7bd547d7557e330a6d4f7a9e1c3766728c2a42374b6cef0d4f02b9" exitCode=0
Nov 01 03:53:56.407795 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.407653    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb" event={"ID":"302961d2-279f-4a93-abcd-264733f897b2","Type":"ContainerDied","Data":"463ebe7dea7bd547d7557e330a6d4f7a9e1c3766728c2a42374b6cef0d4f02b9"}
Nov 01 03:53:56.542409 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.542324    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"]
Nov 01 03:53:56.591463 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.591390    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c476fff-ftssr"]
Nov 01 03:53:56.599126 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.599096    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:53:56.651896 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.651871    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302961d2-279f-4a93-abcd-264733f897b2-kube-api-access-jtz2r\" (UniqueName: \"kubernetes.io/projected/302961d2-279f-4a93-abcd-264733f897b2-kube-api-access-jtz2r\") pod \"302961d2-279f-4a93-abcd-264733f897b2\" (UID: \"302961d2-279f-4a93-abcd-264733f897b2\") "
Nov 01 03:53:56.652020 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.651922    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302961d2-279f-4a93-abcd-264733f897b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302961d2-279f-4a93-abcd-264733f897b2-plugins-config-volume\") pod \"302961d2-279f-4a93-abcd-264733f897b2\" (UID: \"302961d2-279f-4a93-abcd-264733f897b2\") "
Nov 01 03:53:56.652306 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.652284    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302961d2-279f-4a93-abcd-264733f897b2-plugins-config-volume" pod "302961d2-279f-4a93-abcd-264733f897b2" (UID: "302961d2-279f-4a93-abcd-264733f897b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:56.653937 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.653910    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302961d2-279f-4a93-abcd-264733f897b2-kube-api-access-jtz2r" pod "302961d2-279f-4a93-abcd-264733f897b2" (UID: "302961d2-279f-4a93-abcd-264733f897b2"). InnerVolumeSpecName "kube-api-access-jtz2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:56.753747 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.753709    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtz2r\" (UniqueName: \"kubernetes.io/projected/302961d2-279f-4a93-abcd-264733f897b2-kube-api-access-jtz2r\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:56.753747 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:56.753742    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302961d2-279f-4a93-abcd-264733f897b2-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:53:57.415301 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:57.415268    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb" event={"ID":"302961d2-279f-4a93-abcd-264733f897b2","Type":"ContainerDied","Data":"0ecc393f9ca69c384cfbd25d8a1f280f7114d7c46ad3b8c9c876c169d95635ee"}
Nov 01 03:53:57.415301 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:57.415294    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"
Nov 01 03:53:57.415889 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:57.415351    4882 scope.go:122] "RemoveContainer" containerID="463ebe7dea7bd547d7557e330a6d4f7a9e1c3766728c2a42374b6cef0d4f02b9"
Nov 01 03:53:57.589118 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:57.589080    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"]
Nov 01 03:53:57.599864 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:57.599834    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c283a47-e114-4920-a1c0-5344f540ed8d" path="/var/lib/kubelet/pods/4c283a47-e114-4920-a1c0-5344f540ed8d/volumes"
Nov 01 03:53:57.632344 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:57.632307    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3145c214-68b7bb6c77-z99qb"]
Nov 01 03:53:57.842645 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:57.842603    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"]
Nov 01 03:53:57.845218 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:57.845191    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3145c214-lnml6"]
Nov 01 03:53:58.722498 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:53:58.722461    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:53:59.240672 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.240636    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"]
Nov 01 03:53:59.292830 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.292781    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f50c3be9-j9tng"]
Nov 01 03:53:59.336249 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.336213    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"]
Nov 01 03:53:59.337596 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.337559    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy" containerID="cri-o://f562fce218e566b61e5e939e968e1168812d634dd75b34fd379c9108e1ac03f6" gracePeriod=30
Nov 01 03:53:59.612565 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.612484    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="205ee3f8-b63a-4059-b40e-637d85219216" path="/var/lib/kubelet/pods/205ee3f8-b63a-4059-b40e-637d85219216/volumes"
Nov 01 03:53:59.613150 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.613131    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302961d2-279f-4a93-abcd-264733f897b2" path="/var/lib/kubelet/pods/302961d2-279f-4a93-abcd-264733f897b2/volumes"
Nov 01 03:53:59.613653 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.613637    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae2c650-ff58-4caf-ac45-0ca586275770" path="/var/lib/kubelet/pods/aae2c650-ff58-4caf-ac45-0ca586275770/volumes"
Nov 01 03:53:59.653011 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.652984    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"]
Nov 01 03:53:59.653471 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.653457    4882 cpu_manager.go:414] "containerMap: removing container" podUID="302961d2-279f-4a93-abcd-264733f897b2" containerName="payload-processing"
Nov 01 03:53:59.653524 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.653471    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302961d2-279f-4a93-abcd-264733f897b2" containerName="payload-processing"
Nov 01 03:53:59.653524 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.653488    4882 cpu_manager.go:414] "containerMap: removing container" podUID="733da0a6-4de6-4e23-b2be-bc85b00fe9c0" containerName="payload-pre-processing"
Nov 01 03:53:59.653524 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.653493    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733da0a6-4de6-4e23-b2be-bc85b00fe9c0" containerName="payload-pre-processing"
Nov 01 03:53:59.653524 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.653500    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e8a9b728-6638-4a5f-8b86-99a3ed2ca971" containerName="revoke-keys"
Nov 01 03:53:59.653524 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.653505    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a9b728-6638-4a5f-8b86-99a3ed2ca971" containerName="revoke-keys"
Nov 01 03:53:59.653679 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.653591    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="302961d2-279f-4a93-abcd-264733f897b2" containerName="payload-processing"
Nov 01 03:53:59.653679 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.653601    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="733da0a6-4de6-4e23-b2be-bc85b00fe9c0" containerName="payload-pre-processing"
Nov 01 03:53:59.653679 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.653607    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a9b728-6638-4a5f-8b86-99a3ed2ca971" containerName="revoke-keys"
Nov 01 03:53:59.658322 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.658304    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:53:59.715065 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.715039    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:53:59.715238 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.715222    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 03:53:59.715388 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.715358    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:53:59.715518 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.715495    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:53:59.715706 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.715656    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmpns\" (UniqueName: \"kubernetes.io/projected/1ecbc340-f4d4-481d-9127-b7860a49421e-kube-api-access-mmpns\") pod \"maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:53:59.719212 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.719190    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"]
Nov 01 03:53:59.817038 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.816905    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:53:59.817038 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.816971    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:53:59.817038 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.817007    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmpns\" (UniqueName: \"kubernetes.io/projected/1ecbc340-f4d4-481d-9127-b7860a49421e-kube-api-access-mmpns\") pod \"maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:53:59.818057 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.818034    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:53:59.819817 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.819796    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:53:59.820422 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.820398    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"]
Nov 01 03:53:59.825233 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.825213    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:53:59.874484 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.874382    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmpns\" (UniqueName: \"kubernetes.io/projected/1ecbc340-f4d4-481d-9127-b7860a49421e-kube-api-access-mmpns\") pod \"maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:53:59.881195 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.881164    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"]
Nov 01 03:53:59.918288 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.918254    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:53:59.918475 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.918296    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81be89f4-ebde-4d7b-8e24-18bd4a385824-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:53:59.918475 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.918319    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnsv6\" (UniqueName: \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-kube-api-access-gnsv6\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:53:59.918475 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.918342    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:53:59.918656 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.918483    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:53:59.918656 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.918553    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:53:59.918656 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.918585    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:53:59.918656 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.918628    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:53:59.918656 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.918651    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:53:59.926326 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.926305    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-162191-openshift-default-dockercfg-c4j58\""
Nov 01 03:53:59.996883 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:53:59.996839    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:54:00.019324 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.019298    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.019482 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.019367    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.019482 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.019409    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.019612 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.019480    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.019612 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.019508    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81be89f4-ebde-4d7b-8e24-18bd4a385824-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.019612 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.019541    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnsv6\" (UniqueName: \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-kube-api-access-gnsv6\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.019612 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.019571    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.019810 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.019621    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.019810 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.019738    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.020758 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.020127    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.020758 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.020680    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.020912 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.020796    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81be89f4-ebde-4d7b-8e24-18bd4a385824-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.020982 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.020959    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.021126 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.021101    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.023058 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.023030    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.023563 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.023542    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.078602 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.078531    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnsv6\" (UniqueName: \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-kube-api-access-gnsv6\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.078901 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.078864    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.142456 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.142369    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:00.375866 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.375839    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"]
Nov 01 03:54:00.378787 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:54:00.378755    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecbc340_f4d4_481d_9127_b7860a49421e.slice/crio-cbe1d507e846f90aa1ca90b10bef5b2b8fe4d68d9ed7c4f0f7be6af887f2d149 WatchSource:0}: Error finding container cbe1d507e846f90aa1ca90b10bef5b2b8fe4d68d9ed7c4f0f7be6af887f2d149: Status 404 returned error can't find the container with id cbe1d507e846f90aa1ca90b10bef5b2b8fe4d68d9ed7c4f0f7be6af887f2d149
Nov 01 03:54:00.436786 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.436756    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq" event={"ID":"1ecbc340-f4d4-481d-9127-b7860a49421e","Type":"ContainerStarted","Data":"cbe1d507e846f90aa1ca90b10bef5b2b8fe4d68d9ed7c4f0f7be6af887f2d149"}
Nov 01 03:54:00.520270 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.520245    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"]
Nov 01 03:54:00.524278 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:54:00.524252    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81be89f4_ebde_4d7b_8e24_18bd4a385824.slice/crio-4d68925f6318c4527d83ff65df38a42967d099c03a20432964e67b8836ecfcd2 WatchSource:0}: Error finding container 4d68925f6318c4527d83ff65df38a42967d099c03a20432964e67b8836ecfcd2: Status 404 returned error can't find the container with id 4d68925f6318c4527d83ff65df38a42967d099c03a20432964e67b8836ecfcd2
Nov 01 03:54:00.527065 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.527031    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:00.527161 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.527097    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:00.527161 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:00.527127    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:01.443995 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:01.443960    4882 generic.go:356] "Generic (PLEG): container finished" podID="1ecbc340-f4d4-481d-9127-b7860a49421e" containerID="2ca2546ef529200bbf761afd1a3f8b664ddf1f1d8e09928790a91b40822c2d01" exitCode=0
Nov 01 03:54:01.444560 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:01.444298    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq" event={"ID":"1ecbc340-f4d4-481d-9127-b7860a49421e","Type":"ContainerDied","Data":"2ca2546ef529200bbf761afd1a3f8b664ddf1f1d8e09928790a91b40822c2d01"}
Nov 01 03:54:01.446275 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:01.446240    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" event={"ID":"81be89f4-ebde-4d7b-8e24-18bd4a385824","Type":"ContainerStarted","Data":"ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8"}
Nov 01 03:54:01.446411 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:01.446296    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" event={"ID":"81be89f4-ebde-4d7b-8e24-18bd4a385824","Type":"ContainerStarted","Data":"4d68925f6318c4527d83ff65df38a42967d099c03a20432964e67b8836ecfcd2"}
Nov 01 03:54:01.668508 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:01.668384    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" podStartSLOduration=2.668365364 podStartE2EDuration="2.668365364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:01.666379971 +0000 UTC m=+710.228412989" watchObservedRunningTime="2026-11-01 03:54:01.668365364 +0000 UTC m=+710.230398383"
Nov 01 03:54:02.143205 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.143171    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:02.148713 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.148681    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:02.452510 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.452470    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:02.453577 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.453555    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:02.612529 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.612500    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:54:02.650273 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.650243    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-openshift-ca\") pod \"1ecbc340-f4d4-481d-9127-b7860a49421e\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") "
Nov 01 03:54:02.650412 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.650298    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-xks-ca\") pod \"1ecbc340-f4d4-481d-9127-b7860a49421e\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") "
Nov 01 03:54:02.650412 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.650342    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecbc340-f4d4-481d-9127-b7860a49421e-kube-api-access-mmpns\" (UniqueName: \"kubernetes.io/projected/1ecbc340-f4d4-481d-9127-b7860a49421e-kube-api-access-mmpns\") pod \"1ecbc340-f4d4-481d-9127-b7860a49421e\" (UID: \"1ecbc340-f4d4-481d-9127-b7860a49421e\") "
Nov 01 03:54:02.650675 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.650650    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-openshift-ca" pod "1ecbc340-f4d4-481d-9127-b7860a49421e" (UID: "1ecbc340-f4d4-481d-9127-b7860a49421e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:02.650772 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.650756    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:02.652563 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.652535    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecbc340-f4d4-481d-9127-b7860a49421e-kube-api-access-mmpns" pod "1ecbc340-f4d4-481d-9127-b7860a49421e" (UID: "1ecbc340-f4d4-481d-9127-b7860a49421e"). InnerVolumeSpecName "kube-api-access-mmpns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:02.652668 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.652639    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-xks-ca" pod "1ecbc340-f4d4-481d-9127-b7860a49421e" (UID: "1ecbc340-f4d4-481d-9127-b7860a49421e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:54:02.754239 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.754122    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecbc340-f4d4-481d-9127-b7860a49421e-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:02.754239 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:02.754165    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmpns\" (UniqueName: \"kubernetes.io/projected/1ecbc340-f4d4-481d-9127-b7860a49421e-kube-api-access-mmpns\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:03.458881 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:03.458795    4882 generic.go:356] "Generic (PLEG): container finished" podID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerID="b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff" exitCode=0
Nov 01 03:54:03.458881 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:03.458863    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" event={"ID":"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de","Type":"ContainerDied","Data":"b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff"}
Nov 01 03:54:03.459421 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:03.459281    4882 scope.go:122] "RemoveContainer" containerID="b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff"
Nov 01 03:54:03.460300 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:03.460279    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq" event={"ID":"1ecbc340-f4d4-481d-9127-b7860a49421e","Type":"ContainerDied","Data":"cbe1d507e846f90aa1ca90b10bef5b2b8fe4d68d9ed7c4f0f7be6af887f2d149"}
Nov 01 03:54:03.460378 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:03.460328    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbe1d507e846f90aa1ca90b10bef5b2b8fe4d68d9ed7c4f0f7be6af887f2d149"
Nov 01 03:54:03.460495 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:03.460479    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"
Nov 01 03:54:03.460545 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:03.460520    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:03.460619 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:03.460604    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:03.460655 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:03.460643    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:04.492318 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.492267    4882 generic.go:356] "Generic (PLEG): container finished" podID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerID="f562fce218e566b61e5e939e968e1168812d634dd75b34fd379c9108e1ac03f6" exitCode=0
Nov 01 03:54:04.492318 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.492301    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" event={"ID":"7ff0ff2b-a199-46db-a090-b563f32e38b0","Type":"ContainerDied","Data":"f562fce218e566b61e5e939e968e1168812d634dd75b34fd379c9108e1ac03f6"}
Nov 01 03:54:04.492852 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.492396    4882 scope.go:122] "RemoveContainer" containerID="f561a44c87ec7931b7f39406b95dcf99162fabb538838a4f70521c3123890b79"
Nov 01 03:54:04.495717 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.495687    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" event={"ID":"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de","Type":"ContainerStarted","Data":"24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0"}
Nov 01 03:54:04.655875 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.655851    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:54:04.796756 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.796723    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-kube-api-access-gc7lk\" (UniqueName: \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-kube-api-access-gc7lk\") pod \"7ff0ff2b-a199-46db-a090-b563f32e38b0\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") "
Nov 01 03:54:04.796958 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.796791    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff0ff2b-a199-46db-a090-b563f32e38b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff0ff2b-a199-46db-a090-b563f32e38b0-istiod-ca-cert\") pod \"7ff0ff2b-a199-46db-a090-b563f32e38b0\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") "
Nov 01 03:54:04.796958 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.796847    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-socket\") pod \"7ff0ff2b-a199-46db-a090-b563f32e38b0\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") "
Nov 01 03:54:04.796958 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.796931    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-credential-socket\") pod \"7ff0ff2b-a199-46db-a090-b563f32e38b0\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") "
Nov 01 03:54:04.797142 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.796960    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-envoy\") pod \"7ff0ff2b-a199-46db-a090-b563f32e38b0\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") "
Nov 01 03:54:04.797142 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.797006    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-podinfo\") pod \"7ff0ff2b-a199-46db-a090-b563f32e38b0\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") "
Nov 01 03:54:04.797142 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.797058    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-token\" (UniqueName: \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-token\") pod \"7ff0ff2b-a199-46db-a090-b563f32e38b0\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") "
Nov 01 03:54:04.797142 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.797088    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-data\") pod \"7ff0ff2b-a199-46db-a090-b563f32e38b0\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") "
Nov 01 03:54:04.797142 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.797124    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-certs\") pod \"7ff0ff2b-a199-46db-a090-b563f32e38b0\" (UID: \"7ff0ff2b-a199-46db-a090-b563f32e38b0\") "
Nov 01 03:54:04.797670 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.797645    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-credential-socket" pod "7ff0ff2b-a199-46db-a090-b563f32e38b0" (UID: "7ff0ff2b-a199-46db-a090-b563f32e38b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:04.797798 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.797773    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-certs" pod "7ff0ff2b-a199-46db-a090-b563f32e38b0" (UID: "7ff0ff2b-a199-46db-a090-b563f32e38b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:04.798006 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.797983    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-data" pod "7ff0ff2b-a199-46db-a090-b563f32e38b0" (UID: "7ff0ff2b-a199-46db-a090-b563f32e38b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:04.798236 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.798211    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-socket" pod "7ff0ff2b-a199-46db-a090-b563f32e38b0" (UID: "7ff0ff2b-a199-46db-a090-b563f32e38b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:04.798582 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.798228    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff0ff2b-a199-46db-a090-b563f32e38b0-istiod-ca-cert" pod "7ff0ff2b-a199-46db-a090-b563f32e38b0" (UID: "7ff0ff2b-a199-46db-a090-b563f32e38b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:04.799826 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.799804    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-token" pod "7ff0ff2b-a199-46db-a090-b563f32e38b0" (UID: "7ff0ff2b-a199-46db-a090-b563f32e38b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:04.800263 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.800234    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-kube-api-access-gc7lk" pod "7ff0ff2b-a199-46db-a090-b563f32e38b0" (UID: "7ff0ff2b-a199-46db-a090-b563f32e38b0"). InnerVolumeSpecName "kube-api-access-gc7lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:04.800263 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.800247    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-envoy" pod "7ff0ff2b-a199-46db-a090-b563f32e38b0" (UID: "7ff0ff2b-a199-46db-a090-b563f32e38b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:04.802711 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.802692    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-podinfo" pod "7ff0ff2b-a199-46db-a090-b563f32e38b0" (UID: "7ff0ff2b-a199-46db-a090-b563f32e38b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:54:04.898113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.898027    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:04.898113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.898061    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:04.898113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.898071    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:04.898113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.898080    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:04.898113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.898088    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:04.898113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.898096    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:04.898113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.898104    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff0ff2b-a199-46db-a090-b563f32e38b0-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:04.898113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.898112    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc7lk\" (UniqueName: \"kubernetes.io/projected/7ff0ff2b-a199-46db-a090-b563f32e38b0-kube-api-access-gc7lk\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:04.898113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:04.898122    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff0ff2b-a199-46db-a090-b563f32e38b0-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:05.281452 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:05.280760    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:54:05.286453 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:05.284529    4882 patch_prober.go:29] interesting pod/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 03:54:05.286453 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:05.284604    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 03:54:05.503153 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:05.503113    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc" event={"ID":"7ff0ff2b-a199-46db-a090-b563f32e38b0","Type":"ContainerDied","Data":"3106493d922ac4cda8359e76be9768ff1f31e5c988130f84e7750a9577ced243"}
Nov 01 03:54:05.503638 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:05.503178    4882 scope.go:122] "RemoveContainer" containerID="f562fce218e566b61e5e939e968e1168812d634dd75b34fd379c9108e1ac03f6"
Nov 01 03:54:05.503638 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:05.503133    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"
Nov 01 03:54:05.702816 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:05.702784    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"]
Nov 01 03:54:05.748603 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:05.748573    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f50c3be9-gw-openshift-default-7c6b995665-ww4lc"]
Nov 01 03:54:05.942741 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:05.942696    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:54:06.285449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:06.284850    4882 patch_prober.go:29] interesting pod/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 03:54:06.285449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:06.284920    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 03:54:07.281453 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:07.279804    4882 patch_prober.go:29] interesting pod/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 03:54:07.281453 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:07.279878    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 03:54:07.599578 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:07.599503    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" path="/var/lib/kubelet/pods/7ff0ff2b-a199-46db-a090-b563f32e38b0/volumes"
Nov 01 03:54:08.281452 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:08.280807    4882 patch_prober.go:29] interesting pod/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 03:54:08.281452 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:08.280882    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 03:54:09.280842 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:09.280801    4882 patch_prober.go:29] interesting pod/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 03:54:09.281120 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:09.281091    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 03:54:10.279656 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:10.279625    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:54:10.280752 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:10.280731    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:54:13.426378 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:13.426341    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"]
Nov 01 03:54:13.528587 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:13.528556    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"]
Nov 01 03:54:16.565048 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:16.565008    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"]
Nov 01 03:54:16.612356 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:16.612317    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48d2fd87-s4gpb"]
Nov 01 03:54:16.636466 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:16.636415    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"]
Nov 01 03:54:16.636785 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:16.636711    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy" containerID="cri-o://92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3" gracePeriod=30
Nov 01 03:54:17.598500 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.598467    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a9b728-6638-4a5f-8b86-99a3ed2ca971" path="/var/lib/kubelet/pods/e8a9b728-6638-4a5f-8b86-99a3ed2ca971/volumes"
Nov 01 03:54:17.952486 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.952451    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"]
Nov 01 03:54:17.952936 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.952922    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy"
Nov 01 03:54:17.952936 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.952935    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy"
Nov 01 03:54:17.953036 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.952949    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy"
Nov 01 03:54:17.953036 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.952955    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy"
Nov 01 03:54:17.953036 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.952986    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1ecbc340-f4d4-481d-9127-b7860a49421e" containerName="revoke-keys"
Nov 01 03:54:17.953036 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.952992    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ecbc340-f4d4-481d-9127-b7860a49421e" containerName="revoke-keys"
Nov 01 03:54:17.953188 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.953065    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ecbc340-f4d4-481d-9127-b7860a49421e" containerName="revoke-keys"
Nov 01 03:54:17.953188 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.953080    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy"
Nov 01 03:54:17.953188 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.953091    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff0ff2b-a199-46db-a090-b563f32e38b0" containerName="istio-proxy"
Nov 01 03:54:17.960027 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:17.960004    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.016452 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.016186    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-865066-openshift-default-dockercfg-97xzq\""
Nov 01 03:54:18.058141 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.058105    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-token\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.058312 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.058169    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-envoy\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.058312 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.058270    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istiod-ca-cert\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.058395 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.058331    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.058395 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.058361    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-certs\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.058521 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.058400    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-podinfo\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.058573 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.058524    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-data\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.058621 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.058578    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2rl\" (UniqueName: \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-kube-api-access-2k2rl\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.058677 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.058650    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-credential-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.060456 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.060415    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"]
Nov 01 03:54:18.160108 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160075    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-data\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160108 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160123    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2rl\" (UniqueName: \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-kube-api-access-2k2rl\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160335 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160141    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-credential-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160335 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160188    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-token\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160335 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160227    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-envoy\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160335 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160279    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istiod-ca-cert\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160581 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160335    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160581 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160368    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-certs\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160581 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160420    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-podinfo\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160581 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160549    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-data\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160800 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160640    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-credential-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160800 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160750    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-certs\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.160906 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.160842    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.161042 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.161025    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istiod-ca-cert\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.162643 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.162625    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-envoy\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.162844 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.162829    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-podinfo\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.219557 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.219483    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2rl\" (UniqueName: \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-kube-api-access-2k2rl\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.225061 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.225034    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-token\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.273134 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.273105    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:18.426860 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.426820    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5" podUID="48ace860-7548-4aac-a0bf-ee667bb36d1c" containerName="payload-processing" containerID="cri-o://ef81ca08d3c7398b03b86187991c092557cdfad031990ad3989fb8d22d617ba6" gracePeriod=25
Nov 01 03:54:18.529599 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.529518    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls" podUID="fcf77613-8732-4696-9cde-80e5da94e4e9" containerName="payload-pre-processing" containerID="cri-o://af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646" gracePeriod=25
Nov 01 03:54:18.585751 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.585702    4882 generic.go:356] "Generic (PLEG): container finished" podID="48ace860-7548-4aac-a0bf-ee667bb36d1c" containerID="ef81ca08d3c7398b03b86187991c092557cdfad031990ad3989fb8d22d617ba6" exitCode=0
Nov 01 03:54:18.585751 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.585745    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5" event={"ID":"48ace860-7548-4aac-a0bf-ee667bb36d1c","Type":"ContainerDied","Data":"ef81ca08d3c7398b03b86187991c092557cdfad031990ad3989fb8d22d617ba6"}
Nov 01 03:54:18.667358 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.667331    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:54:18.697277 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.697250    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"]
Nov 01 03:54:18.699819 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:54:18.699789    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34faf4a_79c3_4f04_928b_dce7aa9fed52.slice/crio-15f6eccfd21693928384a7d0e161a2bea6c8a20281916db194d348a3a0b14274 WatchSource:0}: Error finding container 15f6eccfd21693928384a7d0e161a2bea6c8a20281916db194d348a3a0b14274: Status 404 returned error can't find the container with id 15f6eccfd21693928384a7d0e161a2bea6c8a20281916db194d348a3a0b14274
Nov 01 03:54:18.702196 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.702163    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:18.702265 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.702230    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:18.702265 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.702258    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:18.766999 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.766606    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ace860-7548-4aac-a0bf-ee667bb36d1c-kube-api-access-dwgdj\" (UniqueName: \"kubernetes.io/projected/48ace860-7548-4aac-a0bf-ee667bb36d1c-kube-api-access-dwgdj\") pod \"48ace860-7548-4aac-a0bf-ee667bb36d1c\" (UID: \"48ace860-7548-4aac-a0bf-ee667bb36d1c\") "
Nov 01 03:54:18.766999 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.766776    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ace860-7548-4aac-a0bf-ee667bb36d1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ace860-7548-4aac-a0bf-ee667bb36d1c-plugins-config-volume\") pod \"48ace860-7548-4aac-a0bf-ee667bb36d1c\" (UID: \"48ace860-7548-4aac-a0bf-ee667bb36d1c\") "
Nov 01 03:54:18.767357 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.767329    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ace860-7548-4aac-a0bf-ee667bb36d1c-plugins-config-volume" pod "48ace860-7548-4aac-a0bf-ee667bb36d1c" (UID: "48ace860-7548-4aac-a0bf-ee667bb36d1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:18.769086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.769057    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ace860-7548-4aac-a0bf-ee667bb36d1c-kube-api-access-dwgdj" pod "48ace860-7548-4aac-a0bf-ee667bb36d1c" (UID: "48ace860-7548-4aac-a0bf-ee667bb36d1c"). InnerVolumeSpecName "kube-api-access-dwgdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:18.788178 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.788150    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:54:18.868200 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.868161    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcf77613-8732-4696-9cde-80e5da94e4e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf77613-8732-4696-9cde-80e5da94e4e9-plugins-config-volume\") pod \"fcf77613-8732-4696-9cde-80e5da94e4e9\" (UID: \"fcf77613-8732-4696-9cde-80e5da94e4e9\") "
Nov 01 03:54:18.868381 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.868264    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf77613-8732-4696-9cde-80e5da94e4e9-kube-api-access-kxh25\" (UniqueName: \"kubernetes.io/projected/fcf77613-8732-4696-9cde-80e5da94e4e9-kube-api-access-kxh25\") pod \"fcf77613-8732-4696-9cde-80e5da94e4e9\" (UID: \"fcf77613-8732-4696-9cde-80e5da94e4e9\") "
Nov 01 03:54:18.868636 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.868608    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf77613-8732-4696-9cde-80e5da94e4e9-plugins-config-volume" pod "fcf77613-8732-4696-9cde-80e5da94e4e9" (UID: "fcf77613-8732-4696-9cde-80e5da94e4e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:18.868700 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.868676    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ace860-7548-4aac-a0bf-ee667bb36d1c-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:18.868700 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.868696    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwgdj\" (UniqueName: \"kubernetes.io/projected/48ace860-7548-4aac-a0bf-ee667bb36d1c-kube-api-access-dwgdj\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:18.870723 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.870698    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf77613-8732-4696-9cde-80e5da94e4e9-kube-api-access-kxh25" pod "fcf77613-8732-4696-9cde-80e5da94e4e9" (UID: "fcf77613-8732-4696-9cde-80e5da94e4e9"). InnerVolumeSpecName "kube-api-access-kxh25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:18.970209 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.970169    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxh25\" (UniqueName: \"kubernetes.io/projected/fcf77613-8732-4696-9cde-80e5da94e4e9-kube-api-access-kxh25\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:18.970209 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:18.970207    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcf77613-8732-4696-9cde-80e5da94e4e9-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:19.593527 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.593498    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"
Nov 01 03:54:19.595128 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.595095    4882 generic.go:356] "Generic (PLEG): container finished" podID="fcf77613-8732-4696-9cde-80e5da94e4e9" containerID="af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646" exitCode=0
Nov 01 03:54:19.595284 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.595244    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"
Nov 01 03:54:19.603466 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.603440    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5" event={"ID":"48ace860-7548-4aac-a0bf-ee667bb36d1c","Type":"ContainerDied","Data":"282b917ed7e6d88ef3ea57a523cdcd0b7038047e7d75d0c219b304a7266afc5a"}
Nov 01 03:54:19.603598 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.603489    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls" event={"ID":"fcf77613-8732-4696-9cde-80e5da94e4e9","Type":"ContainerDied","Data":"af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646"}
Nov 01 03:54:19.603598 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.603519    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls" event={"ID":"fcf77613-8732-4696-9cde-80e5da94e4e9","Type":"ContainerDied","Data":"43eb6e41dd80798a6513e202c46e0422bfdb3ccb6944079361b06cc89c0223cb"}
Nov 01 03:54:19.603598 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.603541    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" event={"ID":"b34faf4a-79c3-4f04-928b-dce7aa9fed52","Type":"ContainerStarted","Data":"eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9"}
Nov 01 03:54:19.603598 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.603564    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" event={"ID":"b34faf4a-79c3-4f04-928b-dce7aa9fed52","Type":"ContainerStarted","Data":"15f6eccfd21693928384a7d0e161a2bea6c8a20281916db194d348a3a0b14274"}
Nov 01 03:54:19.603598 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.603595    4882 scope.go:122] "RemoveContainer" containerID="ef81ca08d3c7398b03b86187991c092557cdfad031990ad3989fb8d22d617ba6"
Nov 01 03:54:19.620346 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.620326    4882 scope.go:122] "RemoveContainer" containerID="af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646"
Nov 01 03:54:19.628789 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.628772    4882 scope.go:122] "RemoveContainer" containerID="af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646"
Nov 01 03:54:19.629068 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:54:19.629047    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646\": container with ID starting with af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646 not found: ID does not exist" containerID="af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646"
Nov 01 03:54:19.629127 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.629075    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646"} err="failed to get container status \"af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646\": rpc error: code = NotFound desc = could not find container \"af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646\": container with ID starting with af230f86e29c3b542136a3439857242ad992690248fdf63c7eee928e43498646 not found: ID does not exist"
Nov 01 03:54:19.746223 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.746190    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"]
Nov 01 03:54:19.792014 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.791979    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6800c12d-cc975bc59-8q8ls"]
Nov 01 03:54:19.893528 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.893451    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"]
Nov 01 03:54:19.900007 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:19.899982    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6800c12d-6875c7b86c-snnq5"]
Nov 01 03:54:20.008957 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:20.008911    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" podStartSLOduration=3.008898702 podStartE2EDuration="3.008898702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:20.008019221 +0000 UTC m=+728.570052238" watchObservedRunningTime="2026-11-01 03:54:20.008898702 +0000 UTC m=+728.570931720"
Nov 01 03:54:20.086088 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:20.086054    4882 patch_prober.go:29] interesting pod/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:54:20.086244 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:20.086107    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:54:20.274155 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:20.274122    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:20.279384 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:20.279361    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:20.606883 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:20.606805    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:20.607820 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:20.607802    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:54:21.599179 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:21.599150    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ace860-7548-4aac-a0bf-ee667bb36d1c" path="/var/lib/kubelet/pods/48ace860-7548-4aac-a0bf-ee667bb36d1c/volumes"
Nov 01 03:54:21.599586 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:21.599521    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcf77613-8732-4696-9cde-80e5da94e4e9" path="/var/lib/kubelet/pods/fcf77613-8732-4696-9cde-80e5da94e4e9/volumes"
Nov 01 03:54:21.900048 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:21.900020    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:54:22.001246 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.001219    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-data\") pod \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") "
Nov 01 03:54:22.001411 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.001255    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-podinfo\") pod \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") "
Nov 01 03:54:22.001411 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.001287    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-token\" (UniqueName: \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-token\") pod \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") "
Nov 01 03:54:22.001411 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.001319    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-kube-api-access-pbgrb\" (UniqueName: \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-kube-api-access-pbgrb\") pod \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") "
Nov 01 03:54:22.001411 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.001383    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-certs\") pod \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") "
Nov 01 03:54:22.001655 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.001472    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-envoy\") pod \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") "
Nov 01 03:54:22.001655 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.001617    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-data" pod "e45c023d-f0c7-4350-9e31-29eabeaa6f11" (UID: "e45c023d-f0c7-4350-9e31-29eabeaa6f11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.001769 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.001627    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istiod-ca-cert\") pod \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") "
Nov 01 03:54:22.001769 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.001701    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-certs" pod "e45c023d-f0c7-4350-9e31-29eabeaa6f11" (UID: "e45c023d-f0c7-4350-9e31-29eabeaa6f11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.001769 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.001710    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-credential-socket\") pod \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") "
Nov 01 03:54:22.001769 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.001766    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-socket\") pod \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\" (UID: \"e45c023d-f0c7-4350-9e31-29eabeaa6f11\") "
Nov 01 03:54:22.002038 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.002011    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-socket" pod "e45c023d-f0c7-4350-9e31-29eabeaa6f11" (UID: "e45c023d-f0c7-4350-9e31-29eabeaa6f11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.002128 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.002106    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istiod-ca-cert" pod "e45c023d-f0c7-4350-9e31-29eabeaa6f11" (UID: "e45c023d-f0c7-4350-9e31-29eabeaa6f11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:22.002302 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.002279    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.002366 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.002305    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.002366 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.002321    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.002366 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.002335    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.002366 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.002342    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-credential-socket" pod "e45c023d-f0c7-4350-9e31-29eabeaa6f11" (UID: "e45c023d-f0c7-4350-9e31-29eabeaa6f11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.003631 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.003603    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-kube-api-access-pbgrb" pod "e45c023d-f0c7-4350-9e31-29eabeaa6f11" (UID: "e45c023d-f0c7-4350-9e31-29eabeaa6f11"). InnerVolumeSpecName "kube-api-access-pbgrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.003746 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.003709    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-token" pod "e45c023d-f0c7-4350-9e31-29eabeaa6f11" (UID: "e45c023d-f0c7-4350-9e31-29eabeaa6f11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.003801 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.003750    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-podinfo" pod "e45c023d-f0c7-4350-9e31-29eabeaa6f11" (UID: "e45c023d-f0c7-4350-9e31-29eabeaa6f11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:54:22.004037 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.004017    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-envoy" pod "e45c023d-f0c7-4350-9e31-29eabeaa6f11" (UID: "e45c023d-f0c7-4350-9e31-29eabeaa6f11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.103574 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.103548    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.103574 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.103574    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45c023d-f0c7-4350-9e31-29eabeaa6f11-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.103750 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.103584    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.103750 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.103595    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.103750 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.103606    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbgrb\" (UniqueName: \"kubernetes.io/projected/e45c023d-f0c7-4350-9e31-29eabeaa6f11-kube-api-access-pbgrb\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.619490 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.619455    4882 generic.go:356] "Generic (PLEG): container finished" podID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerID="92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3" exitCode=0
Nov 01 03:54:22.619941 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.619497    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"
Nov 01 03:54:22.619941 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.619533    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" event={"ID":"e45c023d-f0c7-4350-9e31-29eabeaa6f11","Type":"ContainerDied","Data":"92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3"}
Nov 01 03:54:22.619941 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.619596    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2" event={"ID":"e45c023d-f0c7-4350-9e31-29eabeaa6f11","Type":"ContainerDied","Data":"625cc2d8f09f0d2f4e1ab9d86f34e3f2321ac7c99fe1c12a8edc66bf577bcff8"}
Nov 01 03:54:22.619941 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.619621    4882 scope.go:122] "RemoveContainer" containerID="92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3"
Nov 01 03:54:22.629719 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.629695    4882 scope.go:122] "RemoveContainer" containerID="d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528"
Nov 01 03:54:22.637815 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.637796    4882 scope.go:122] "RemoveContainer" containerID="92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3"
Nov 01 03:54:22.638065 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:54:22.638044    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3\": container with ID starting with 92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3 not found: ID does not exist" containerID="92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3"
Nov 01 03:54:22.638114 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.638073    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3"} err="failed to get container status \"92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3\": rpc error: code = NotFound desc = could not find container \"92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3\": container with ID starting with 92d9d13498e4433d67829eaab43d80eb01b7d96dc829a9eb3a973cb564c9f5d3 not found: ID does not exist"
Nov 01 03:54:22.638114 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.638089    4882 scope.go:122] "RemoveContainer" containerID="d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528"
Nov 01 03:54:22.638319 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:54:22.638297    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528\": container with ID starting with d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528 not found: ID does not exist" containerID="d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528"
Nov 01 03:54:22.638372 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.638328    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528"} err="failed to get container status \"d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528\": rpc error: code = NotFound desc = could not find container \"d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528\": container with ID starting with d22e2637293599f5a40798b3250cbc5eaf5edf543fa4acae9ba6c54353dcb528 not found: ID does not exist"
Nov 01 03:54:22.820465 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.820416    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"]
Nov 01 03:54:22.866423 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:22.866393    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-48d2fd87-openshift-default-6dfc86c667-b7bm2"]
Nov 01 03:54:23.598748 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:23.598716    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" path="/var/lib/kubelet/pods/e45c023d-f0c7-4350-9e31-29eabeaa6f11/volumes"
Nov 01 03:54:28.721723 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:54:28.721687    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:54:38.750884 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:38.750747    4882 generic.go:356] "Generic (PLEG): container finished" podID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerID="ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8" exitCode=0
Nov 01 03:54:38.752108 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:38.751000    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" event={"ID":"81be89f4-ebde-4d7b-8e24-18bd4a385824","Type":"ContainerDied","Data":"ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8"}
Nov 01 03:54:38.753096 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:38.753065    4882 scope.go:122] "RemoveContainer" containerID="ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8"
Nov 01 03:54:38.754669 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:38.754636    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:38.754782 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:38.754741    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:38.754844 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:38.754791    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:39.736864 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:39.736826    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"]
Nov 01 03:54:39.759014 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:39.758981    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" event={"ID":"81be89f4-ebde-4d7b-8e24-18bd4a385824","Type":"ContainerStarted","Data":"f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b"}
Nov 01 03:54:39.792496 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:39.791726    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6800c12d-5f4cq"]
Nov 01 03:54:39.837453 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:39.837409    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"]
Nov 01 03:54:39.837627 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:39.837606    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy" containerID="cri-o://24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0" gracePeriod=30
Nov 01 03:54:40.143108 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:40.143023    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:40.143259 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:40.143204    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 03:54:40.143259 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:40.143248    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 03:54:40.286832 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:40.286786    4882 patch_prober.go:29] interesting pod/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:54:40.287008 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:40.286871    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:54:41.143176 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.143143    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 03:54:41.143583 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.143204    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 03:54:41.252242 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252209    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"]
Nov 01 03:54:41.252676 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252662    4882 cpu_manager.go:414] "containerMap: removing container" podUID="fcf77613-8732-4696-9cde-80e5da94e4e9" containerName="payload-pre-processing"
Nov 01 03:54:41.252725 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252677    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf77613-8732-4696-9cde-80e5da94e4e9" containerName="payload-pre-processing"
Nov 01 03:54:41.252725 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252690    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy"
Nov 01 03:54:41.252725 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252695    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy"
Nov 01 03:54:41.252725 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252712    4882 cpu_manager.go:414] "containerMap: removing container" podUID="48ace860-7548-4aac-a0bf-ee667bb36d1c" containerName="payload-processing"
Nov 01 03:54:41.252725 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252716    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ace860-7548-4aac-a0bf-ee667bb36d1c" containerName="payload-processing"
Nov 01 03:54:41.252888 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252737    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy"
Nov 01 03:54:41.252888 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252743    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy"
Nov 01 03:54:41.252888 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252817    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy"
Nov 01 03:54:41.252888 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252828    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf77613-8732-4696-9cde-80e5da94e4e9" containerName="payload-pre-processing"
Nov 01 03:54:41.252888 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.252836    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ace860-7548-4aac-a0bf-ee667bb36d1c" containerName="payload-processing"
Nov 01 03:54:41.256180 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.256158    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.305245 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.305216    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c2c259-openshift-default-dockercfg-7n2f9\""
Nov 01 03:54:41.323560 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.323526    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.323560 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.323558    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.323737 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.323644    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.323737 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.323714    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.323807 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.323739    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.323841 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.323803    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.323884 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.323865    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.323929 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.323916    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.323966 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.323936    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66hg5\" (UniqueName: \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-kube-api-access-66hg5\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.355491 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.355468    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"]
Nov 01 03:54:41.424845 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.424765    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.424845 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.424805    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.424845 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.424823    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.425074 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.424856    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.425074 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.424878    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.425074 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.424910    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.425074 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.424983    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66hg5\" (UniqueName: \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-kube-api-access-66hg5\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.425253 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.425112    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.425253 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.425139    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.425253 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.425224    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.425412 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.425256    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.425412 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.425320    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.425554 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.425484    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.425819 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.425800    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.427386 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.427362    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.427495 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.427404    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.489257 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.489225    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.489671 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.489645    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hg5\" (UniqueName: \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-kube-api-access-66hg5\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.613167 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.612858    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:41.621833 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:41.621801    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ecbc340-f4d4-481d-9127-b7860a49421e" path="/var/lib/kubelet/pods/1ecbc340-f4d4-481d-9127-b7860a49421e/volumes"
Nov 01 03:54:42.057279 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:42.057255    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"]
Nov 01 03:54:42.059620 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:54:42.059589    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc94dbdc_bb4f_4b4b_aa94_a516adfb041f.slice/crio-e54b9a5a16a4763c1c6a560af0f098dd2dfc4826b98d9e663eaafb9237178e2b WatchSource:0}: Error finding container e54b9a5a16a4763c1c6a560af0f098dd2dfc4826b98d9e663eaafb9237178e2b: Status 404 returned error can't find the container with id e54b9a5a16a4763c1c6a560af0f098dd2dfc4826b98d9e663eaafb9237178e2b
Nov 01 03:54:42.071446 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:42.062091    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:42.071669 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:42.071637    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:42.081514 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:42.071801    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:42.143498 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:42.143474    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 03:54:42.143810 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:42.143518    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 03:54:42.801733 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:42.801689    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" event={"ID":"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f","Type":"ContainerStarted","Data":"91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d"}
Nov 01 03:54:42.801891 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:42.801760    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" event={"ID":"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f","Type":"ContainerStarted","Data":"e54b9a5a16a4763c1c6a560af0f098dd2dfc4826b98d9e663eaafb9237178e2b"}
Nov 01 03:54:42.995454 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:42.986411    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" podStartSLOduration=1.98639164 podStartE2EDuration="1.98639164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:42.960569502 +0000 UTC m=+751.522602520" watchObservedRunningTime="2026-11-01 03:54:42.98639164 +0000 UTC m=+751.548424658"
Nov 01 03:54:43.143595 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:43.143508    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 03:54:43.143595 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:43.143567    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 03:54:43.613989 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:43.613953    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:43.619871 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:43.619845    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:43.808001 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:43.807966    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:43.808783 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:43.808759    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:54:44.143621 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:44.143587    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 03:54:44.144070 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:44.143641    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 03:54:45.122942 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.122919    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:54:45.146804 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.146781    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:45.147167 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.147064    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:45.147879 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.147860    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 03:54:45.160198 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160174    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-envoy\") pod \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") "
Nov 01 03:54:45.160297 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160215    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-credential-socket\") pod \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") "
Nov 01 03:54:45.160297 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160246    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-kube-api-access-grvbq\" (UniqueName: \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-kube-api-access-grvbq\") pod \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") "
Nov 01 03:54:45.160297 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160270    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-socket\") pod \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") "
Nov 01 03:54:45.160475 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160302    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-podinfo\") pod \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") "
Nov 01 03:54:45.160475 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160411    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istiod-ca-cert\") pod \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") "
Nov 01 03:54:45.160590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160485    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-data\") pod \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") "
Nov 01 03:54:45.160590 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160522    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-token\" (UniqueName: \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-token\") pod \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") "
Nov 01 03:54:45.160688 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160596    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-certs\") pod \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\" (UID: \"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de\") "
Nov 01 03:54:45.160688 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160655    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-credential-socket" pod "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" (UID: "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:45.160688 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160667    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-socket" pod "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" (UID: "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:45.160871 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160847    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-data" pod "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" (UID: "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:45.160984 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160958    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:45.160984 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160970    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-certs" pod "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" (UID: "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:45.161128 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.160986    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:45.161128 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.161002    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:45.161128 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.161079    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istiod-ca-cert" pod "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" (UID: "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:45.162614 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.162586    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-podinfo" pod "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" (UID: "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:54:45.162703 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.162643    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-token" pod "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" (UID: "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:45.162779 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.162763    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-envoy" pod "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" (UID: "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:45.162824 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.162779    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-kube-api-access-grvbq" pod "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" (UID: "0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de"). InnerVolumeSpecName "kube-api-access-grvbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:45.261911 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.261872    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:45.261911 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.261898    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:45.261911 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.261911    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grvbq\" (UniqueName: \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-kube-api-access-grvbq\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:45.262196 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.261926    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:45.262196 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.261941    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:45.262196 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.261954    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:54:45.822257 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.822222    4882 generic.go:356] "Generic (PLEG): container finished" podID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerID="24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0" exitCode=0
Nov 01 03:54:45.822439 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.822273    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"
Nov 01 03:54:45.822439 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.822308    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" event={"ID":"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de","Type":"ContainerDied","Data":"24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0"}
Nov 01 03:54:45.822439 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.822366    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr" event={"ID":"0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de","Type":"ContainerDied","Data":"2ff3a312b9203bc88c5b4287447af5013edd0bc8ec6c41828718821fbdcf9aa3"}
Nov 01 03:54:45.822439 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.822388    4882 scope.go:122] "RemoveContainer" containerID="24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0"
Nov 01 03:54:45.831555 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.831539    4882 scope.go:122] "RemoveContainer" containerID="b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff"
Nov 01 03:54:45.839705 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.839689    4882 scope.go:122] "RemoveContainer" containerID="24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0"
Nov 01 03:54:45.839978 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:54:45.839953    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0\": container with ID starting with 24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0 not found: ID does not exist" containerID="24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0"
Nov 01 03:54:45.840072 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.839982    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0"} err="failed to get container status \"24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0\": rpc error: code = NotFound desc = could not find container \"24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0\": container with ID starting with 24635ed33dd44ddeadaa14b3ba477837dacf241728de647661adda0c9552afc0 not found: ID does not exist"
Nov 01 03:54:45.840072 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.839998    4882 scope.go:122] "RemoveContainer" containerID="b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff"
Nov 01 03:54:45.840247 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:54:45.840226    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff\": container with ID starting with b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff not found: ID does not exist" containerID="b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff"
Nov 01 03:54:45.840284 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:45.840254    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff"} err="failed to get container status \"b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff\": rpc error: code = NotFound desc = could not find container \"b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff\": container with ID starting with b1f734d520cdfdadf6ac2862715470f054c1a1783498c99d24ebf103a618baff not found: ID does not exist"
Nov 01 03:54:46.023561 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:46.023527    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"]
Nov 01 03:54:46.069491 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:46.069463    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6800c12d-openshift-default-6b7978bf6f-tslfr"]
Nov 01 03:54:47.599243 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:47.599212    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" path="/var/lib/kubelet/pods/0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de/volumes"
Nov 01 03:54:53.582251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.582174    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"]
Nov 01 03:54:53.582670 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.582655    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy"
Nov 01 03:54:53.582714 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.582671    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy"
Nov 01 03:54:53.582714 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.582683    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy"
Nov 01 03:54:53.582714 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.582688    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy"
Nov 01 03:54:53.582806 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.582761    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45c023d-f0c7-4350-9e31-29eabeaa6f11" containerName="istio-proxy"
Nov 01 03:54:53.582806 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.582770    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy"
Nov 01 03:54:53.587484 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.587464    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.643451 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.640809    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a9c5b3\"/\"default-dockercfg-fvxkt\""
Nov 01 03:54:53.665093 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.665060    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"]
Nov 01 03:54:53.666836 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.666800    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b62aaf-4775-4a62-a49c-c2ada60bea08-tls-certs\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.666983 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.666889    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-tmp-dir\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.667164 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.667144    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-dshm\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.667223 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.667177    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-model-cache\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.667287 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.667255    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rxp7\" (UniqueName: \"kubernetes.io/projected/28b62aaf-4775-4a62-a49c-c2ada60bea08-kube-api-access-6rxp7\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.667360 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.667346    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-home\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.687030 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.686993    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a9c5b3\"/\"simulator-w4-a9c5b3-kserve-self-signed-certs\""
Nov 01 03:54:53.687030 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.687023    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a9c5b3\"/\"kube-root-ca.crt\""
Nov 01 03:54:53.687224 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.687060    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a9c5b3\"/\"openshift-service-ca.crt\""
Nov 01 03:54:53.768891 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.768854    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-home\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.769080 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.768925    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b62aaf-4775-4a62-a49c-c2ada60bea08-tls-certs\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.769080 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.768982    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-tmp-dir\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.769080 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.769059    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-dshm\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.769251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.769084    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-model-cache\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.769251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.769148    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rxp7\" (UniqueName: \"kubernetes.io/projected/28b62aaf-4775-4a62-a49c-c2ada60bea08-kube-api-access-6rxp7\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.769362 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.769333    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-home\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.769419 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.769374    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-tmp-dir\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.769546 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.769524    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-model-cache\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.771385 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.771359    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-dshm\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.771511 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.771396    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b62aaf-4775-4a62-a49c-c2ada60bea08-tls-certs\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.826485 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.826453    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rxp7\" (UniqueName: \"kubernetes.io/projected/28b62aaf-4775-4a62-a49c-c2ada60bea08-kube-api-access-6rxp7\") pod \"simulator-w4-a9c5b3-kserve-75df979996-jspgh\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") " pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:53.900958 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:53.900875    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:54.275508 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:54.275483    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"]
Nov 01 03:54:54.277786 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:54:54.277761    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28b62aaf_4775_4a62_a49c_c2ada60bea08.slice/crio-e701b70041cb4302e239f1dbac6ab17e8a353f18abf9c396257546f15db24617 WatchSource:0}: Error finding container e701b70041cb4302e239f1dbac6ab17e8a353f18abf9c396257546f15db24617: Status 404 returned error can't find the container with id e701b70041cb4302e239f1dbac6ab17e8a353f18abf9c396257546f15db24617
Nov 01 03:54:54.889099 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:54.889065    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" event={"ID":"28b62aaf-4775-4a62-a49c-c2ada60bea08","Type":"ContainerStarted","Data":"96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b"}
Nov 01 03:54:54.889522 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:54.889107    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" event={"ID":"28b62aaf-4775-4a62-a49c-c2ada60bea08","Type":"ContainerStarted","Data":"e701b70041cb4302e239f1dbac6ab17e8a353f18abf9c396257546f15db24617"}
Nov 01 03:54:54.906573 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:54.906537    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:54:54.999840 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:54:54.999786    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podStartSLOduration=1.9997711219999998 podStartE2EDuration="1.999771122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:54.999555332 +0000 UTC m=+763.561588367" watchObservedRunningTime="2026-11-01 03:54:54.999771122 +0000 UTC m=+763.561804171"
Nov 01 03:54:58.723845 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:54:58.723810    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:55:03.901303 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:03.901226    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:55:03.913968 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:03.913938    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:55:03.940521 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:03.940492    4882 generic.go:356] "Generic (PLEG): container finished" podID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerID="eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9" exitCode=0
Nov 01 03:55:03.940690 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:03.940569    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" event={"ID":"b34faf4a-79c3-4f04-928b-dce7aa9fed52","Type":"ContainerDied","Data":"eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9"}
Nov 01 03:55:03.941057 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:03.941039    4882 scope.go:122] "RemoveContainer" containerID="eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9"
Nov 01 03:55:03.942351 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:03.942319    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:55:03.942410 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:03.942383    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:55:03.942468 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:03.942415    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:55:03.952396 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:03.952377    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 03:55:04.948959 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:04.948926    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" event={"ID":"b34faf4a-79c3-4f04-928b-dce7aa9fed52","Type":"ContainerStarted","Data":"589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b"}
Nov 01 03:55:05.275805 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:05.275713    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:55:05.276140 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:05.276117    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 03:55:05.276296 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:05.276273    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 03:55:05.607867 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:05.607788    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:55:06.283451 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:06.282854    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 03:55:06.283451 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:06.282920    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 03:55:07.277452 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:07.273844    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 03:55:07.277452 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:07.273905    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 03:55:08.282450 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:08.281904    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 03:55:08.282450 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:08.281962    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 03:55:09.285452 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:09.278815    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 03:55:09.285452 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:09.278876    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 03:55:10.277353 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:10.277321    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:55:10.278274 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:10.278251    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 03:55:25.327678 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.327641    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"]
Nov 01 03:55:25.328250 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.328204    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7ad2d2-f9d6-49fc-b80f-2e6ad8ae85de" containerName="istio-proxy"
Nov 01 03:55:25.331298 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.331279    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.387633 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.387585    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"openshift-service-ca.crt\""
Nov 01 03:55:25.416708 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.416674    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcnwb\" (UniqueName: \"kubernetes.io/projected/a6635a62-77fe-4864-9430-c372b90d2e85-kube-api-access-pcnwb\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.416861 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.416743    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-model-cache\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.416861 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.416830    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-tmp-dir\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.416934 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.416860    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6635a62-77fe-4864-9430-c372b90d2e85-tls-certs\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.416979 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.416961    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-dshm\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.417013 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.417003    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-home\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.427278 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.427259    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"kube-root-ca.crt\""
Nov 01 03:55:25.427376 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.427287    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"default-dockercfg-qwscg\""
Nov 01 03:55:25.427376 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.427317    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"simulator-w3-f73b9f-kserve-self-signed-certs\""
Nov 01 03:55:25.429640 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.429620    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"]
Nov 01 03:55:25.518637 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.518594    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-tmp-dir\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.518637 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.518638    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6635a62-77fe-4864-9430-c372b90d2e85-tls-certs\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.518897 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.518689    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-dshm\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.518897 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.518714    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-home\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.518897 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.518785    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcnwb\" (UniqueName: \"kubernetes.io/projected/a6635a62-77fe-4864-9430-c372b90d2e85-kube-api-access-pcnwb\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.518897 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.518855    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-model-cache\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.519098 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.519022    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-tmp-dir\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.519171 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.519145    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-home\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.519226 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.519190    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-model-cache\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.521107 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.521085    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-dshm\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.521260 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.521244    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6635a62-77fe-4864-9430-c372b90d2e85-tls-certs\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.588854 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.588780    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcnwb\" (UniqueName: \"kubernetes.io/projected/a6635a62-77fe-4864-9430-c372b90d2e85-kube-api-access-pcnwb\") pod \"simulator-w3-f73b9f-kserve-5d96b65797-smhf2\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") " pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:25.642660 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:25.642630    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:26.040664 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:26.040637    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"]
Nov 01 03:55:26.043196 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:55:26.043165    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6635a62_77fe_4864_9430_c372b90d2e85.slice/crio-8460fec174d06d365abb399199227ad77f5c1d3c0a0fea79163597ee057612c5 WatchSource:0}: Error finding container 8460fec174d06d365abb399199227ad77f5c1d3c0a0fea79163597ee057612c5: Status 404 returned error can't find the container with id 8460fec174d06d365abb399199227ad77f5c1d3c0a0fea79163597ee057612c5
Nov 01 03:55:26.113802 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:26.113771    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" event={"ID":"a6635a62-77fe-4864-9430-c372b90d2e85","Type":"ContainerStarted","Data":"58fc46b431fa03455d050ba6e6831976682041dcd02581e5ca7ce65a1dbaee43"}
Nov 01 03:55:26.113929 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:26.113813    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" event={"ID":"a6635a62-77fe-4864-9430-c372b90d2e85","Type":"ContainerStarted","Data":"8460fec174d06d365abb399199227ad77f5c1d3c0a0fea79163597ee057612c5"}
Nov 01 03:55:26.224347 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:26.224292    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podStartSLOduration=1.224276196 podStartE2EDuration="1.224276196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:55:26.22278257 +0000 UTC m=+794.784815588" watchObservedRunningTime="2026-11-01 03:55:26.224276196 +0000 UTC m=+794.786309243"
Nov 01 03:55:26.643486 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:26.643416    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:28.723346 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:55:28.723308    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:55:29.128385 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:29.126314    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": read tcp 10.132.0.2:42102->10.132.0.70:15021: read: connection reset by peer" start-of-body=
Nov 01 03:55:29.128385 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:29.126371    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": read tcp 10.132.0.2:42102->10.132.0.70:15021: read: connection reset by peer"
Nov 01 03:55:30.139472 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:30.139413    4882 generic.go:356] "Generic (PLEG): container finished" podID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerID="91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d" exitCode=0
Nov 01 03:55:30.139865 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:30.139485    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" event={"ID":"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f","Type":"ContainerDied","Data":"91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d"}
Nov 01 03:55:30.139865 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:30.139850    4882 scope.go:122] "RemoveContainer" containerID="91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d"
Nov 01 03:55:30.141178 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:30.141155    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:55:30.141228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:30.141215    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:55:30.141262 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:30.141243    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:55:31.149580 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:31.149536    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" event={"ID":"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f","Type":"ContainerStarted","Data":"8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a"}
Nov 01 03:55:31.613812 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:31.613778    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:55:31.614009 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:31.613930    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 03:55:31.614009 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:31.613979    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 03:55:32.614316 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:32.614277    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 03:55:32.614728 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:32.614337    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 03:55:33.614802 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:33.614770    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 03:55:33.615194 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:33.614822    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 03:55:34.613868 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:34.613831    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 03:55:34.614045 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:34.613894    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 03:55:35.614229 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:35.614195    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 03:55:35.614646 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:35.614254    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 03:55:35.643235 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:35.643196    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:35.655720 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:35.655691    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:36.219986 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:36.219950    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 03:55:36.617828 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:36.617745    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:55:36.618246 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:36.618205    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:55:36.618840 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:36.618819    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 03:55:50.822419 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.822379    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"]
Nov 01 03:55:50.827289 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.827265    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.894706 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.894674    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-tmp-dir\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.894706 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.894712    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1301538d-17a0-4021-aeda-db3130e114b1-tls-certs\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.894946 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.894728    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtgj\" (UniqueName: \"kubernetes.io/projected/1301538d-17a0-4021-aeda-db3130e114b1-kube-api-access-jjtgj\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.894946 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.894793    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-dshm\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.894946 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.894893    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-model-cache\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.895116 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.894953    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-home\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.929053 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.929020    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-865066\"/\"openshift-service-ca.crt\""
Nov 01 03:55:50.929053 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.929043    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-865066\"/\"default-dockercfg-p58bn\""
Nov 01 03:55:50.929273 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.929187    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-865066\"/\"kube-root-ca.crt\""
Nov 01 03:55:50.929343 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.929314    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-865066\"/\"simulator-w2-865066-kserve-self-signed-certs\""
Nov 01 03:55:50.931661 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.931640    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"]
Nov 01 03:55:50.995936 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.995899    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-home\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.998219 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.996151    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-tmp-dir\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.998219 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.996196    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1301538d-17a0-4021-aeda-db3130e114b1-tls-certs\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.998219 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.996224    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtgj\" (UniqueName: \"kubernetes.io/projected/1301538d-17a0-4021-aeda-db3130e114b1-kube-api-access-jjtgj\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.998219 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.996250    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-dshm\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.998219 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.996406    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-model-cache\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:50.998219 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:50.997518    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-model-cache\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:51.001902 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:51.001878    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-tmp-dir\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:51.005127 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:51.001953    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-home\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:51.005298 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:51.004644    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-dshm\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:51.005298 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:51.005138    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1301538d-17a0-4021-aeda-db3130e114b1-tls-certs\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:51.057567 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:51.057534    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtgj\" (UniqueName: \"kubernetes.io/projected/1301538d-17a0-4021-aeda-db3130e114b1-kube-api-access-jjtgj\") pod \"simulator-w2-865066-kserve-54d996bff6-8crkc\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") " pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:51.139241 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:51.139170    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:51.505953 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:51.505924    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"]
Nov 01 03:55:51.508817 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:55:51.508788    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1301538d_17a0_4021_aeda_db3130e114b1.slice/crio-7329c7ef8511329cc16826feb6e8499fcb0f1ed5cb8233a58ef7bba6a16f3d80 WatchSource:0}: Error finding container 7329c7ef8511329cc16826feb6e8499fcb0f1ed5cb8233a58ef7bba6a16f3d80: Status 404 returned error can't find the container with id 7329c7ef8511329cc16826feb6e8499fcb0f1ed5cb8233a58ef7bba6a16f3d80
Nov 01 03:55:52.291259 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:52.291221    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" event={"ID":"1301538d-17a0-4021-aeda-db3130e114b1","Type":"ContainerStarted","Data":"b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196"}
Nov 01 03:55:52.291651 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:52.291272    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" event={"ID":"1301538d-17a0-4021-aeda-db3130e114b1","Type":"ContainerStarted","Data":"7329c7ef8511329cc16826feb6e8499fcb0f1ed5cb8233a58ef7bba6a16f3d80"}
Nov 01 03:55:52.400719 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:52.400659    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" podStartSLOduration=2.400647454 podStartE2EDuration="2.400647454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:55:52.400037165 +0000 UTC m=+820.962070182" watchObservedRunningTime="2026-11-01 03:55:52.400647454 +0000 UTC m=+820.962680471"
Nov 01 03:55:53.139378 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:55:53.139342    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:55:58.722703 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:55:58.722664    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:56:01.139646 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:01.139608    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:56:01.152565 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:01.152534    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:56:01.355905 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:01.355875    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 03:56:24.403948 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.403875    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"]
Nov 01 03:56:24.408763 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.408742    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.457830 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.457797    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-home\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.457987 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.457850    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkq6p\" (UniqueName: \"kubernetes.io/projected/ca3d592c-9b95-465e-ae21-f5f9417eb20f-kube-api-access-fkq6p\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.457987 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.457926    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-dshm\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.458128 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.458108    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tmp-dir\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.458217 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.458199    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-model-cache\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.458278 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.458235    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tls-certs\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.459970 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.459948    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c2c259\"/\"openshift-service-ca.crt\""
Nov 01 03:56:24.505840 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.505816    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c2c259\"/\"kube-root-ca.crt\""
Nov 01 03:56:24.505931 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.505902    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c2c259\"/\"default-dockercfg-pfmj4\""
Nov 01 03:56:24.506033 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.506014    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c2c259\"/\"simulator-w1-c2c259-kserve-self-signed-certs\""
Nov 01 03:56:24.508358 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.508341    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"]
Nov 01 03:56:24.559672 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.559642    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tmp-dir\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.559846 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.559700    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-model-cache\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.559893 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.559860    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tls-certs\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.559950 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.559935    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-home\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.559985 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.559965    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkq6p\" (UniqueName: \"kubernetes.io/projected/ca3d592c-9b95-465e-ae21-f5f9417eb20f-kube-api-access-fkq6p\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.560021 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.559985    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-dshm\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.560063 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.560030    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-model-cache\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.560116 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.560097    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tmp-dir\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.560290 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.560268    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-home\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.562156 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.562133    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-dshm\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.562351 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.562336    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tls-certs\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.616792 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.616766    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkq6p\" (UniqueName: \"kubernetes.io/projected/ca3d592c-9b95-465e-ae21-f5f9417eb20f-kube-api-access-fkq6p\") pod \"simulator-w1-c2c259-kserve-578cf77794-26dc8\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") " pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:24.747218 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:24.747188    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:25.113238 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:25.113215    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"]
Nov 01 03:56:25.115660 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:56:25.115621    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca3d592c_9b95_465e_ae21_f5f9417eb20f.slice/crio-9a9f625a73ecb922e94d4ad3791315b2b96b2297dffd24983536f36bd58bd201 WatchSource:0}: Error finding container 9a9f625a73ecb922e94d4ad3791315b2b96b2297dffd24983536f36bd58bd201: Status 404 returned error can't find the container with id 9a9f625a73ecb922e94d4ad3791315b2b96b2297dffd24983536f36bd58bd201
Nov 01 03:56:25.498932 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:25.498897    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" event={"ID":"ca3d592c-9b95-465e-ae21-f5f9417eb20f","Type":"ContainerStarted","Data":"5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30"}
Nov 01 03:56:25.498932 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:25.498936    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" event={"ID":"ca3d592c-9b95-465e-ae21-f5f9417eb20f","Type":"ContainerStarted","Data":"9a9f625a73ecb922e94d4ad3791315b2b96b2297dffd24983536f36bd58bd201"}
Nov 01 03:56:25.608145 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:25.608088    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podStartSLOduration=1.608071788 podStartE2EDuration="1.608071788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:56:25.607285895 +0000 UTC m=+854.169318913" watchObservedRunningTime="2026-11-01 03:56:25.608071788 +0000 UTC m=+854.170104806"
Nov 01 03:56:25.747884 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:25.747850    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:28.722568 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:56:28.722531    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:56:34.748267 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:34.748228    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:34.760933 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:34.760911    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:35.570235 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:35.570206    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 03:56:43.008562 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.008525    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"]
Nov 01 03:56:43.012509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.012483    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.110316 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.110284    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-865066\"/\"premium-w2-865066-kserve-self-signed-certs\""
Nov 01 03:56:43.112848 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.112824    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"]
Nov 01 03:56:43.173947 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.173914    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-model-cache\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.174119 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.173975    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-home\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.174119 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.174053    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-tmp-dir\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.174119 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.174112    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8104497-11eb-49d7-b2b5-328dda72bb22-tls-certs\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.174227 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.174180    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-dshm\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.174261 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.174244    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xckcx\" (UniqueName: \"kubernetes.io/projected/f8104497-11eb-49d7-b2b5-328dda72bb22-kube-api-access-xckcx\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.274942 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.274858    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-model-cache\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.274942 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.274929    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-home\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.275147 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.274956    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-tmp-dir\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.275147 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.275021    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8104497-11eb-49d7-b2b5-328dda72bb22-tls-certs\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.275147 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.275055    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-dshm\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.275147 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.275126    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xckcx\" (UniqueName: \"kubernetes.io/projected/f8104497-11eb-49d7-b2b5-328dda72bb22-kube-api-access-xckcx\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.275369 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.275346    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-model-cache\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.275511 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.275489    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-tmp-dir\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.275551 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.275492    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-home\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.277441 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.277399    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-dshm\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.277650 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.277634    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8104497-11eb-49d7-b2b5-328dda72bb22-tls-certs\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.332052 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.332020    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xckcx\" (UniqueName: \"kubernetes.io/projected/f8104497-11eb-49d7-b2b5-328dda72bb22-kube-api-access-xckcx\") pod \"premium-w2-865066-kserve-fd656859c-h8thg\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") " pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.624818 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.624744    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:43.992636 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:43.992609    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"]
Nov 01 03:56:43.994853 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:56:43.994824    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8104497_11eb_49d7_b2b5_328dda72bb22.slice/crio-b7b2e24e3d43988b92eb31e0afbcb0aaa50018265018447ce64f1f23cad2de73 WatchSource:0}: Error finding container b7b2e24e3d43988b92eb31e0afbcb0aaa50018265018447ce64f1f23cad2de73: Status 404 returned error can't find the container with id b7b2e24e3d43988b92eb31e0afbcb0aaa50018265018447ce64f1f23cad2de73
Nov 01 03:56:44.614231 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:44.614191    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" event={"ID":"f8104497-11eb-49d7-b2b5-328dda72bb22","Type":"ContainerStarted","Data":"77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80"}
Nov 01 03:56:44.614231 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:44.614234    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" event={"ID":"f8104497-11eb-49d7-b2b5-328dda72bb22","Type":"ContainerStarted","Data":"b7b2e24e3d43988b92eb31e0afbcb0aaa50018265018447ce64f1f23cad2de73"}
Nov 01 03:56:44.625498 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:44.625475    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:44.723076 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:44.722988    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podStartSLOduration=2.722971331 podStartE2EDuration="2.722971331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:56:44.722788451 +0000 UTC m=+873.284821470" watchObservedRunningTime="2026-11-01 03:56:44.722971331 +0000 UTC m=+873.285004341"
Nov 01 03:56:53.625950 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:53.625918    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:53.638074 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:53.638052    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:53.677514 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:56:53.677489    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 03:56:58.722164 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:56:58.722125    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:57:02.117418 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.117383    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"]
Nov 01 03:57:02.120926 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.120907    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.169482 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.169451    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f73b9f-openshift-default-dockercfg-rtg6h\""
Nov 01 03:57:02.178962 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.178921    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.179064 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.178975    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgswm\" (UniqueName: \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-kube-api-access-zgswm\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.179113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.179087    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.179157 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.179132    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.179214 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.179153    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.179251 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.179233    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.179283 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.179258    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.179317 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.179297    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.179415 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.179397    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.217973 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.217943    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"]
Nov 01 03:57:02.280651 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.280615    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.280818 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.280670    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.280818 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.280689    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.280818 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.280707    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.280818 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.280738    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.280818 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.280781    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.281139 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.280835    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.281139 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.280879    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.281139 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.280923    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgswm\" (UniqueName: \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-kube-api-access-zgswm\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.281139 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.281102    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.281293 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.281151    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.281293 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.281180    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.281402 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.281339    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.281616 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.281596    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.283235 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.283214    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.283325 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.283270    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.337808 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.337772    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.337980 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.337908    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgswm\" (UniqueName: \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-kube-api-access-zgswm\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.435012 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.434977    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:02.858284 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.858261    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"]
Nov 01 03:57:02.860362 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:57:02.860309    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a77365_4bae_4d90_95c9_43f6eb75d25a.slice/crio-0bd84773eff1fc37f9d60c6f553f96e55a9112554c58a1112aaa8a487f30a970 WatchSource:0}: Error finding container 0bd84773eff1fc37f9d60c6f553f96e55a9112554c58a1112aaa8a487f30a970: Status 404 returned error can't find the container with id 0bd84773eff1fc37f9d60c6f553f96e55a9112554c58a1112aaa8a487f30a970
Nov 01 03:57:02.862815 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.862785    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:57:02.862874 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.862851    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:57:02.862907 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:02.862878    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:57:03.723581 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:03.723542    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" event={"ID":"a7a77365-4bae-4d90-95c9-43f6eb75d25a","Type":"ContainerStarted","Data":"dce86ae9a8d77a60fe019005018bf304f2af3e76b6406d692ebb9f2d066737f0"}
Nov 01 03:57:03.723978 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:03.723629    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" event={"ID":"a7a77365-4bae-4d90-95c9-43f6eb75d25a","Type":"ContainerStarted","Data":"0bd84773eff1fc37f9d60c6f553f96e55a9112554c58a1112aaa8a487f30a970"}
Nov 01 03:57:03.938587 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:03.938527    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" podStartSLOduration=1.9385088700000002 podStartE2EDuration="1.93850887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:57:03.89162399 +0000 UTC m=+892.453657003" watchObservedRunningTime="2026-11-01 03:57:03.93850887 +0000 UTC m=+892.500541893"
Nov 01 03:57:04.435345 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:04.435310    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:04.436706 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:04.436683    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 03:57:04.436782 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:04.436729    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" podUID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 03:57:05.435494 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:05.435456    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 03:57:05.435951 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:05.435509    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" podUID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 03:57:06.436034 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:06.435996    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 03:57:06.436413 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:06.436060    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" podUID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 03:57:07.436145 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:07.436103    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 03:57:07.436647 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:07.436179    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" podUID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 03:57:08.435863 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:08.435827    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 03:57:08.436021 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:08.435890    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" podUID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 03:57:09.427160 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:09.427119    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"]
Nov 01 03:57:09.427668 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:09.427374    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy" containerID="cri-o://d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6" gracePeriod=30
Nov 01 03:57:09.440888 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:09.440865    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:09.441153 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:09.441135    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:09.441854 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:09.441835    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 03:57:11.694812 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:11.694782    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:57:11.696866 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:11.696842    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 03:57:14.690381 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.690356    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:57:14.789980 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.789895    4882 generic.go:356] "Generic (PLEG): container finished" podID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerID="d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6" exitCode=0
Nov 01 03:57:14.789980 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.789950    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"
Nov 01 03:57:14.790180 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.789982    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" event={"ID":"732dba6f-5c7f-48bf-8802-6742d16cc51d","Type":"ContainerDied","Data":"d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6"}
Nov 01 03:57:14.790180 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.790036    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz" event={"ID":"732dba6f-5c7f-48bf-8802-6742d16cc51d","Type":"ContainerDied","Data":"832bd65b5eb3ba83318c7d0d63921fa2424b166d6a6f8902f7888bbb0f5cfb21"}
Nov 01 03:57:14.790180 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.790068    4882 scope.go:122] "RemoveContainer" containerID="d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6"
Nov 01 03:57:14.798915 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.798897    4882 scope.go:122] "RemoveContainer" containerID="000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5"
Nov 01 03:57:14.807524 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.807507    4882 scope.go:122] "RemoveContainer" containerID="d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6"
Nov 01 03:57:14.807798 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:57:14.807782    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6\": container with ID starting with d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6 not found: ID does not exist" containerID="d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6"
Nov 01 03:57:14.807845 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.807807    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6"} err="failed to get container status \"d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6\": rpc error: code = NotFound desc = could not find container \"d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6\": container with ID starting with d2602b355c8e450a46854a20711a62d3e406ebd3c6110f08127806b54b8395d6 not found: ID does not exist"
Nov 01 03:57:14.807845 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.807825    4882 scope.go:122] "RemoveContainer" containerID="000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5"
Nov 01 03:57:14.808030 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:57:14.808016    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5\": container with ID starting with 000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5 not found: ID does not exist" containerID="000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5"
Nov 01 03:57:14.808069 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.808034    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5"} err="failed to get container status \"000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5\": rpc error: code = NotFound desc = could not find container \"000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5\": container with ID starting with 000b3a36008e7d69ee2cb5dc299cb8768ec3eec20f5236aae2b0b498ae8a8dd5 not found: ID does not exist"
Nov 01 03:57:14.814414 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.814392    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-token\" (UniqueName: \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-token\") pod \"732dba6f-5c7f-48bf-8802-6742d16cc51d\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") "
Nov 01 03:57:14.814524 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.814469    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-credential-socket\") pod \"732dba6f-5c7f-48bf-8802-6742d16cc51d\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") "
Nov 01 03:57:14.814524 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.814488    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-kube-api-access-ch8vx\" (UniqueName: \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-kube-api-access-ch8vx\") pod \"732dba6f-5c7f-48bf-8802-6742d16cc51d\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") "
Nov 01 03:57:14.814638 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.814550    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-data\") pod \"732dba6f-5c7f-48bf-8802-6742d16cc51d\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") "
Nov 01 03:57:14.814638 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.814587    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-podinfo\") pod \"732dba6f-5c7f-48bf-8802-6742d16cc51d\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") "
Nov 01 03:57:14.814742 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.814655    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-certs\") pod \"732dba6f-5c7f-48bf-8802-6742d16cc51d\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") "
Nov 01 03:57:14.814742 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.814682    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-envoy\") pod \"732dba6f-5c7f-48bf-8802-6742d16cc51d\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") "
Nov 01 03:57:14.814742 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.814730    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-socket\") pod \"732dba6f-5c7f-48bf-8802-6742d16cc51d\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") "
Nov 01 03:57:14.814897 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.814759    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732dba6f-5c7f-48bf-8802-6742d16cc51d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732dba6f-5c7f-48bf-8802-6742d16cc51d-istiod-ca-cert\") pod \"732dba6f-5c7f-48bf-8802-6742d16cc51d\" (UID: \"732dba6f-5c7f-48bf-8802-6742d16cc51d\") "
Nov 01 03:57:14.814897 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.814790    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-credential-socket" pod "732dba6f-5c7f-48bf-8802-6742d16cc51d" (UID: "732dba6f-5c7f-48bf-8802-6742d16cc51d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:57:14.814897 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.814876    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-data" pod "732dba6f-5c7f-48bf-8802-6742d16cc51d" (UID: "732dba6f-5c7f-48bf-8802-6742d16cc51d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:57:14.815083 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.815061    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-socket" pod "732dba6f-5c7f-48bf-8802-6742d16cc51d" (UID: "732dba6f-5c7f-48bf-8802-6742d16cc51d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:57:14.815193 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.815172    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732dba6f-5c7f-48bf-8802-6742d16cc51d-istiod-ca-cert" pod "732dba6f-5c7f-48bf-8802-6742d16cc51d" (UID: "732dba6f-5c7f-48bf-8802-6742d16cc51d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:57:14.815301 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.815204    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-certs" pod "732dba6f-5c7f-48bf-8802-6742d16cc51d" (UID: "732dba6f-5c7f-48bf-8802-6742d16cc51d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:57:14.815366 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.815303    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:14.815366 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.815324    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:14.815366 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.815339    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:14.816668 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.816645    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-token" pod "732dba6f-5c7f-48bf-8802-6742d16cc51d" (UID: "732dba6f-5c7f-48bf-8802-6742d16cc51d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:57:14.816854 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.816835    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-envoy" pod "732dba6f-5c7f-48bf-8802-6742d16cc51d" (UID: "732dba6f-5c7f-48bf-8802-6742d16cc51d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:57:14.817055 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.817041    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-podinfo" pod "732dba6f-5c7f-48bf-8802-6742d16cc51d" (UID: "732dba6f-5c7f-48bf-8802-6742d16cc51d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:57:14.817113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.817076    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-kube-api-access-ch8vx" pod "732dba6f-5c7f-48bf-8802-6742d16cc51d" (UID: "732dba6f-5c7f-48bf-8802-6742d16cc51d"). InnerVolumeSpecName "kube-api-access-ch8vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:57:14.916454 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.916399    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:14.916454 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.916439    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch8vx\" (UniqueName: \"kubernetes.io/projected/732dba6f-5c7f-48bf-8802-6742d16cc51d-kube-api-access-ch8vx\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:14.916454 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.916451    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:14.916454 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.916461    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:14.916712 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.916471    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732dba6f-5c7f-48bf-8802-6742d16cc51d-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:14.916712 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:14.916480    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732dba6f-5c7f-48bf-8802-6742d16cc51d-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:15.203759 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:15.203726    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"]
Nov 01 03:57:15.207484 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:15.207455    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-vjrbz"]
Nov 01 03:57:15.599709 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:15.599677    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" path="/var/lib/kubelet/pods/732dba6f-5c7f-48bf-8802-6742d16cc51d/volumes"
Nov 01 03:57:27.099155 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.099119    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"]
Nov 01 03:57:27.099626 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.099611    4882 cpu_manager.go:414] "containerMap: removing container" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy"
Nov 01 03:57:27.099681 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.099628    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy"
Nov 01 03:57:27.099681 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.099655    4882 cpu_manager.go:414] "containerMap: removing container" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy"
Nov 01 03:57:27.099681 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.099660    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy"
Nov 01 03:57:27.099784 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.099752    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy"
Nov 01 03:57:27.102982 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.102961    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:27.159606 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.159569    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"]
Nov 01 03:57:27.200291 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.200253    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:57:27.200501 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.200421    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 03:57:27.266648 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.266611    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d539f4aa-zdzsg\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:27.266826 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.266672    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7lbj\" (UniqueName: \"kubernetes.io/projected/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-kube-api-access-n7lbj\") pod \"maas-api-revoke-sub-d539f4aa-zdzsg\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:27.266902 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.266881    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d539f4aa-zdzsg\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:27.368089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.368003    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d539f4aa-zdzsg\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:27.368089 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.368063    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d539f4aa-zdzsg\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:27.368280 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.368103    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7lbj\" (UniqueName: \"kubernetes.io/projected/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-kube-api-access-n7lbj\") pod \"maas-api-revoke-sub-d539f4aa-zdzsg\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:27.368736 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.368705    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d539f4aa-zdzsg\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:27.370525 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.370506    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d539f4aa-zdzsg\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:27.425682 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.425649    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7lbj\" (UniqueName: \"kubernetes.io/projected/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-kube-api-access-n7lbj\") pod \"maas-api-revoke-sub-d539f4aa-zdzsg\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:27.715660 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:27.715625    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:28.082784 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:28.082757    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"]
Nov 01 03:57:28.085220 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:57:28.085186    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954ca3b4_985a_4a30_b2ae_18cfed6e6e56.slice/crio-40ab7e9060568436a0dcef33268ac57a5cd5bb4fb6ba23c12c42babae17fa30c WatchSource:0}: Error finding container 40ab7e9060568436a0dcef33268ac57a5cd5bb4fb6ba23c12c42babae17fa30c: Status 404 returned error can't find the container with id 40ab7e9060568436a0dcef33268ac57a5cd5bb4fb6ba23c12c42babae17fa30c
Nov 01 03:57:28.722164 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:57:28.722127    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:57:28.877555 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:28.877518    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg" event={"ID":"954ca3b4-985a-4a30-b2ae-18cfed6e6e56","Type":"ContainerStarted","Data":"2a080ab54b2057261177968c922d81b112c2411ead11239ec13be23f7a8a7d7d"}
Nov 01 03:57:28.877733 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:28.877570    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg" event={"ID":"954ca3b4-985a-4a30-b2ae-18cfed6e6e56","Type":"ContainerStarted","Data":"40ab7e9060568436a0dcef33268ac57a5cd5bb4fb6ba23c12c42babae17fa30c"}
Nov 01 03:57:28.987364 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:28.987267    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg" podStartSLOduration=1.987254322 podStartE2EDuration="1.987254322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:57:28.985403548 +0000 UTC m=+917.547436559" watchObservedRunningTime="2026-11-01 03:57:28.987254322 +0000 UTC m=+917.549287339"
Nov 01 03:57:29.884631 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:29.884597    4882 generic.go:356] "Generic (PLEG): container finished" podID="954ca3b4-985a-4a30-b2ae-18cfed6e6e56" containerID="2a080ab54b2057261177968c922d81b112c2411ead11239ec13be23f7a8a7d7d" exitCode=0
Nov 01 03:57:29.885042 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:29.884667    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg" event={"ID":"954ca3b4-985a-4a30-b2ae-18cfed6e6e56","Type":"ContainerDied","Data":"2a080ab54b2057261177968c922d81b112c2411ead11239ec13be23f7a8a7d7d"}
Nov 01 03:57:31.040198 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.040171    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:31.207893 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.207858    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-openshift-ca\") pod \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") "
Nov 01 03:57:31.208062 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.208009    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-kube-api-access-n7lbj\" (UniqueName: \"kubernetes.io/projected/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-kube-api-access-n7lbj\") pod \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") "
Nov 01 03:57:31.208116 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.208101    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-xks-ca\") pod \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\" (UID: \"954ca3b4-985a-4a30-b2ae-18cfed6e6e56\") "
Nov 01 03:57:31.208299 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.208271    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-openshift-ca" pod "954ca3b4-985a-4a30-b2ae-18cfed6e6e56" (UID: "954ca3b4-985a-4a30-b2ae-18cfed6e6e56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:57:31.208496 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.208480    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:31.210171 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.210137    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-xks-ca" pod "954ca3b4-985a-4a30-b2ae-18cfed6e6e56" (UID: "954ca3b4-985a-4a30-b2ae-18cfed6e6e56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:57:31.210260 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.210212    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-kube-api-access-n7lbj" pod "954ca3b4-985a-4a30-b2ae-18cfed6e6e56" (UID: "954ca3b4-985a-4a30-b2ae-18cfed6e6e56"). InnerVolumeSpecName "kube-api-access-n7lbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:57:31.309367 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.309336    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7lbj\" (UniqueName: \"kubernetes.io/projected/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-kube-api-access-n7lbj\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:31.309367 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.309363    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954ca3b4-985a-4a30-b2ae-18cfed6e6e56-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:31.897357 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.897267    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg" event={"ID":"954ca3b4-985a-4a30-b2ae-18cfed6e6e56","Type":"ContainerDied","Data":"40ab7e9060568436a0dcef33268ac57a5cd5bb4fb6ba23c12c42babae17fa30c"}
Nov 01 03:57:31.897357 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.897332    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ab7e9060568436a0dcef33268ac57a5cd5bb4fb6ba23c12c42babae17fa30c"
Nov 01 03:57:31.897357 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:31.897281    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"
Nov 01 03:57:42.409314 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.409273    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"]
Nov 01 03:57:42.409869 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.409773    4882 cpu_manager.go:414] "containerMap: removing container" podUID="954ca3b4-985a-4a30-b2ae-18cfed6e6e56" containerName="revoke-keys"
Nov 01 03:57:42.409869 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.409786    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954ca3b4-985a-4a30-b2ae-18cfed6e6e56" containerName="revoke-keys"
Nov 01 03:57:42.409947 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.409872    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="732dba6f-5c7f-48bf-8802-6742d16cc51d" containerName="istio-proxy"
Nov 01 03:57:42.409947 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.409880    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="954ca3b4-985a-4a30-b2ae-18cfed6e6e56" containerName="revoke-keys"
Nov 01 03:57:42.414338 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.414317    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:42.510898 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.510864    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:57:42.511101 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.510938    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 03:57:42.513471 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.513437    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"]
Nov 01 03:57:42.532414 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.532376    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5437d00e-2dlbc\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:42.532414 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.532412    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwmn\" (UniqueName: \"kubernetes.io/projected/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-kube-api-access-wjwmn\") pod \"maas-api-revoke-sub-5437d00e-2dlbc\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:42.532682 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.532633    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5437d00e-2dlbc\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:42.633944 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.633907    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5437d00e-2dlbc\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:42.633944 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.633946    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwmn\" (UniqueName: \"kubernetes.io/projected/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-kube-api-access-wjwmn\") pod \"maas-api-revoke-sub-5437d00e-2dlbc\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:42.634189 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.634038    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5437d00e-2dlbc\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:42.634775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.634749    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5437d00e-2dlbc\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:42.636442 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.636395    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5437d00e-2dlbc\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:42.691964 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.691937    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwmn\" (UniqueName: \"kubernetes.io/projected/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-kube-api-access-wjwmn\") pod \"maas-api-revoke-sub-5437d00e-2dlbc\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:42.728048 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:42.728011    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:43.106279 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:43.106252    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"]
Nov 01 03:57:43.108593 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:57:43.108566    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6be10b_6de2_4c8c_a0a6_684c9eb5a3ae.slice/crio-bf2d1eee84312947d72c054ddfcf6103f97ddf72036a57eae13318d6521d47ff WatchSource:0}: Error finding container bf2d1eee84312947d72c054ddfcf6103f97ddf72036a57eae13318d6521d47ff: Status 404 returned error can't find the container with id bf2d1eee84312947d72c054ddfcf6103f97ddf72036a57eae13318d6521d47ff
Nov 01 03:57:43.964989 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:43.964946    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc" event={"ID":"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae","Type":"ContainerStarted","Data":"0481eb0ba71faa948630466f2afe90813e723f01427c042e99c783a4feef9ce0"}
Nov 01 03:57:43.965353 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:43.965000    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc" event={"ID":"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae","Type":"ContainerStarted","Data":"bf2d1eee84312947d72c054ddfcf6103f97ddf72036a57eae13318d6521d47ff"}
Nov 01 03:57:44.075440 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:44.075383    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc" podStartSLOduration=2.075368469 podStartE2EDuration="2.075368469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:57:44.073098437 +0000 UTC m=+932.635131488" watchObservedRunningTime="2026-11-01 03:57:44.075368469 +0000 UTC m=+932.637401536"
Nov 01 03:57:44.971996 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:44.971957    4882 generic.go:356] "Generic (PLEG): container finished" podID="8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae" containerID="0481eb0ba71faa948630466f2afe90813e723f01427c042e99c783a4feef9ce0" exitCode=0
Nov 01 03:57:44.972400 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:44.972046    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc" event={"ID":"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae","Type":"ContainerDied","Data":"0481eb0ba71faa948630466f2afe90813e723f01427c042e99c783a4feef9ce0"}
Nov 01 03:57:46.133156 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.133134    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:46.273422 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.273336    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-kube-api-access-wjwmn\" (UniqueName: \"kubernetes.io/projected/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-kube-api-access-wjwmn\") pod \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") "
Nov 01 03:57:46.273422 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.273413    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-xks-ca\") pod \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") "
Nov 01 03:57:46.273666 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.273561    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-openshift-ca\") pod \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\" (UID: \"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae\") "
Nov 01 03:57:46.274298 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.274267    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-openshift-ca" pod "8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae" (UID: "8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:57:46.275716 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.275685    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-kube-api-access-wjwmn" pod "8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae" (UID: "8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae"). InnerVolumeSpecName "kube-api-access-wjwmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:57:46.276203 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.276172    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-xks-ca" pod "8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae" (UID: "8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:57:46.375371 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.375340    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:46.375371 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.375367    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjwmn\" (UniqueName: \"kubernetes.io/projected/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-kube-api-access-wjwmn\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:46.375586 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.375380    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:46.984792 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.984764    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"
Nov 01 03:57:46.984980 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.984796    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc" event={"ID":"8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae","Type":"ContainerDied","Data":"bf2d1eee84312947d72c054ddfcf6103f97ddf72036a57eae13318d6521d47ff"}
Nov 01 03:57:46.984980 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:46.984853    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf2d1eee84312947d72c054ddfcf6103f97ddf72036a57eae13318d6521d47ff"
Nov 01 03:57:54.396802 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.396723    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"]
Nov 01 03:57:54.397245 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.397229    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae" containerName="revoke-keys"
Nov 01 03:57:54.397289 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.397246    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae" containerName="revoke-keys"
Nov 01 03:57:54.397347 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.397338    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae" containerName="revoke-keys"
Nov 01 03:57:54.400894 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.400874    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:54.454637 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.454608    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:57:54.454766 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.454709    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 03:57:54.455235 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.455210    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"]
Nov 01 03:57:54.458860 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.458837    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"]
Nov 01 03:57:54.468501 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.468473    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11615f6e-gt2mm\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:54.468693 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.468536    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11615f6e-gt2mm\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:54.468693 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.468621    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6tjv\" (UniqueName: \"kubernetes.io/projected/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-kube-api-access-z6tjv\") pod \"maas-api-revoke-sub-11615f6e-gt2mm\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:54.473812 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.473789    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:54.517159 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.517130    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"]
Nov 01 03:57:54.569819 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.569782    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btf8l\" (UniqueName: \"kubernetes.io/projected/48fb3d3c-7149-4c11-af83-24eb4efc05d1-kube-api-access-btf8l\") pod \"maas-api-revoke-sub-be0c667b-vjmlp\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:54.570152 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.570130    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0c667b-vjmlp\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:54.570343 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.570325    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0c667b-vjmlp\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:54.570537 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.570520    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11615f6e-gt2mm\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:54.571509 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.571485    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11615f6e-gt2mm\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:54.581448 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.572096    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6tjv\" (UniqueName: \"kubernetes.io/projected/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-kube-api-access-z6tjv\") pod \"maas-api-revoke-sub-11615f6e-gt2mm\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:54.584054 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.571371    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11615f6e-gt2mm\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:54.584205 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.582194    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11615f6e-gt2mm\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:54.645342 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.645309    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6tjv\" (UniqueName: \"kubernetes.io/projected/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-kube-api-access-z6tjv\") pod \"maas-api-revoke-sub-11615f6e-gt2mm\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:54.691495 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.688525    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btf8l\" (UniqueName: \"kubernetes.io/projected/48fb3d3c-7149-4c11-af83-24eb4efc05d1-kube-api-access-btf8l\") pod \"maas-api-revoke-sub-be0c667b-vjmlp\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:54.691495 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.688604    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0c667b-vjmlp\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:54.691495 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.688641    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0c667b-vjmlp\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:54.691495 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.689541    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0c667b-vjmlp\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:54.698448 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.697605    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0c667b-vjmlp\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:54.716626 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.716597    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:54.770578 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.770547    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btf8l\" (UniqueName: \"kubernetes.io/projected/48fb3d3c-7149-4c11-af83-24eb4efc05d1-kube-api-access-btf8l\") pod \"maas-api-revoke-sub-be0c667b-vjmlp\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:54.795451 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:54.794297    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:55.180867 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:55.180777    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"]
Nov 01 03:57:55.219841 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:57:55.219814    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb261d8c5_5fc0_4efa_b929_e8ba71f46e7c.slice/crio-3802691d17971286a9a61d58512866d03d627a17e066f398a27e608999043d4f WatchSource:0}: Error finding container 3802691d17971286a9a61d58512866d03d627a17e066f398a27e608999043d4f: Status 404 returned error can't find the container with id 3802691d17971286a9a61d58512866d03d627a17e066f398a27e608999043d4f
Nov 01 03:57:55.230017 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:55.229995    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"]
Nov 01 03:57:56.041379 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:56.041351    4882 generic.go:356] "Generic (PLEG): container finished" podID="48fb3d3c-7149-4c11-af83-24eb4efc05d1" containerID="0c55af4ffce59a231874ca638962b8aaa08d027068ba6a1cccc9c7ea0474a9da" exitCode=0
Nov 01 03:57:56.041786 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:56.041412    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp" event={"ID":"48fb3d3c-7149-4c11-af83-24eb4efc05d1","Type":"ContainerDied","Data":"0c55af4ffce59a231874ca638962b8aaa08d027068ba6a1cccc9c7ea0474a9da"}
Nov 01 03:57:56.041786 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:56.041520    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp" event={"ID":"48fb3d3c-7149-4c11-af83-24eb4efc05d1","Type":"ContainerStarted","Data":"1e835ffa92f8ebfd64342c86248715e578e5406c681fb55d2913b44a0ac13ffc"}
Nov 01 03:57:56.043179 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:56.043158    4882 generic.go:356] "Generic (PLEG): container finished" podID="b261d8c5-5fc0-4efa-b929-e8ba71f46e7c" containerID="29b05b62507662b2eb0a7adccc5e4bda76a43858ee968e369ce8f8c0f15f76a4" exitCode=0
Nov 01 03:57:56.043279 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:56.043244    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm" event={"ID":"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c","Type":"ContainerDied","Data":"29b05b62507662b2eb0a7adccc5e4bda76a43858ee968e369ce8f8c0f15f76a4"}
Nov 01 03:57:56.043321 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:56.043284    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm" event={"ID":"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c","Type":"ContainerStarted","Data":"3802691d17971286a9a61d58512866d03d627a17e066f398a27e608999043d4f"}
Nov 01 03:57:57.239343 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.239318    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:57.242742 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.242720    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:57.332761 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.332672    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-xks-ca\") pod \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") "
Nov 01 03:57:57.332761 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.332736    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-kube-api-access-z6tjv\" (UniqueName: \"kubernetes.io/projected/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-kube-api-access-z6tjv\") pod \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") "
Nov 01 03:57:57.332990 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.332792    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-xks-ca\") pod \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") "
Nov 01 03:57:57.332990 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.332837    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-openshift-ca\") pod \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\" (UID: \"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c\") "
Nov 01 03:57:57.332990 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.332874    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-openshift-ca\") pod \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") "
Nov 01 03:57:57.333138 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.333069    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fb3d3c-7149-4c11-af83-24eb4efc05d1-kube-api-access-btf8l\" (UniqueName: \"kubernetes.io/projected/48fb3d3c-7149-4c11-af83-24eb4efc05d1-kube-api-access-btf8l\") pod \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\" (UID: \"48fb3d3c-7149-4c11-af83-24eb4efc05d1\") "
Nov 01 03:57:57.333279 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.333248    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-openshift-ca" pod "b261d8c5-5fc0-4efa-b929-e8ba71f46e7c" (UID: "b261d8c5-5fc0-4efa-b929-e8ba71f46e7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:57:57.333334 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.333290    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-openshift-ca" pod "48fb3d3c-7149-4c11-af83-24eb4efc05d1" (UID: "48fb3d3c-7149-4c11-af83-24eb4efc05d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:57:57.333536 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.333511    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:57.333607 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.333541    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:57.334993 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.334967    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-xks-ca" pod "b261d8c5-5fc0-4efa-b929-e8ba71f46e7c" (UID: "b261d8c5-5fc0-4efa-b929-e8ba71f46e7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:57:57.335092 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.334966    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-xks-ca" pod "48fb3d3c-7149-4c11-af83-24eb4efc05d1" (UID: "48fb3d3c-7149-4c11-af83-24eb4efc05d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:57:57.335162 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.335136    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-kube-api-access-z6tjv" pod "b261d8c5-5fc0-4efa-b929-e8ba71f46e7c" (UID: "b261d8c5-5fc0-4efa-b929-e8ba71f46e7c"). InnerVolumeSpecName "kube-api-access-z6tjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:57:57.335282 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.335240    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fb3d3c-7149-4c11-af83-24eb4efc05d1-kube-api-access-btf8l" pod "48fb3d3c-7149-4c11-af83-24eb4efc05d1" (UID: "48fb3d3c-7149-4c11-af83-24eb4efc05d1"). InnerVolumeSpecName "kube-api-access-btf8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:57:57.434535 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.434496    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btf8l\" (UniqueName: \"kubernetes.io/projected/48fb3d3c-7149-4c11-af83-24eb4efc05d1-kube-api-access-btf8l\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:57.434535 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.434535    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:57.434732 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.434552    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6tjv\" (UniqueName: \"kubernetes.io/projected/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c-kube-api-access-z6tjv\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:57.434732 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.434566    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48fb3d3c-7149-4c11-af83-24eb4efc05d1-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:57:57.647515 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.647485    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"]
Nov 01 03:57:57.647958 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.647945    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b261d8c5-5fc0-4efa-b929-e8ba71f46e7c" containerName="revoke-keys"
Nov 01 03:57:57.647999 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.647961    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b261d8c5-5fc0-4efa-b929-e8ba71f46e7c" containerName="revoke-keys"
Nov 01 03:57:57.647999 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.647969    4882 cpu_manager.go:414] "containerMap: removing container" podUID="48fb3d3c-7149-4c11-af83-24eb4efc05d1" containerName="revoke-keys"
Nov 01 03:57:57.647999 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.647975    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48fb3d3c-7149-4c11-af83-24eb4efc05d1" containerName="revoke-keys"
Nov 01 03:57:57.648091 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.648055    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b261d8c5-5fc0-4efa-b929-e8ba71f46e7c" containerName="revoke-keys"
Nov 01 03:57:57.648091 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.648062    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="48fb3d3c-7149-4c11-af83-24eb4efc05d1" containerName="revoke-keys"
Nov 01 03:57:57.651492 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.651472    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:57:57.706594 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.706551    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"]
Nov 01 03:57:57.738668 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.738627    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7t9h\" (UniqueName: \"kubernetes.io/projected/22ab814f-40d0-4ef3-bf2d-0c8029341145-kube-api-access-g7t9h\") pod \"maas-api-revoke-sub-c022b9e0-qrw6b\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:57:57.738839 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.738765    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c022b9e0-qrw6b\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:57:57.739007 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.738938    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c022b9e0-qrw6b\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:57:57.840565 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.840522    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7t9h\" (UniqueName: \"kubernetes.io/projected/22ab814f-40d0-4ef3-bf2d-0c8029341145-kube-api-access-g7t9h\") pod \"maas-api-revoke-sub-c022b9e0-qrw6b\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:57:57.840765 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.840593    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c022b9e0-qrw6b\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:57:57.840765 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.840669    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c022b9e0-qrw6b\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:57:57.841389 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.841363    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c022b9e0-qrw6b\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:57:57.843031 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.843009    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c022b9e0-qrw6b\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:57:57.897538 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.897470    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7t9h\" (UniqueName: \"kubernetes.io/projected/22ab814f-40d0-4ef3-bf2d-0c8029341145-kube-api-access-g7t9h\") pod \"maas-api-revoke-sub-c022b9e0-qrw6b\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:57:57.964877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:57.964843    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:57:58.057555 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:58.057133    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp" event={"ID":"48fb3d3c-7149-4c11-af83-24eb4efc05d1","Type":"ContainerDied","Data":"1e835ffa92f8ebfd64342c86248715e578e5406c681fb55d2913b44a0ac13ffc"}
Nov 01 03:57:58.057555 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:58.057197    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e835ffa92f8ebfd64342c86248715e578e5406c681fb55d2913b44a0ac13ffc"
Nov 01 03:57:58.057555 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:58.057279    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"
Nov 01 03:57:58.062851 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:58.061708    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm" event={"ID":"b261d8c5-5fc0-4efa-b929-e8ba71f46e7c","Type":"ContainerDied","Data":"3802691d17971286a9a61d58512866d03d627a17e066f398a27e608999043d4f"}
Nov 01 03:57:58.062851 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:58.061758    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3802691d17971286a9a61d58512866d03d627a17e066f398a27e608999043d4f"
Nov 01 03:57:58.062851 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:58.061822    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"
Nov 01 03:57:58.331554 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:58.331527    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"]
Nov 01 03:57:58.333592 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:57:58.333551    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ab814f_40d0_4ef3_bf2d_0c8029341145.slice/crio-803fea82170c09e95831353e10feded1c614c9ca11c65bca7ac36f3c465050a6 WatchSource:0}: Error finding container 803fea82170c09e95831353e10feded1c614c9ca11c65bca7ac36f3c465050a6: Status 404 returned error can't find the container with id 803fea82170c09e95831353e10feded1c614c9ca11c65bca7ac36f3c465050a6
Nov 01 03:57:58.722557 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:57:58.722526    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:57:59.070144 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:59.070108    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b" event={"ID":"22ab814f-40d0-4ef3-bf2d-0c8029341145","Type":"ContainerStarted","Data":"afede98c939c3d38671842c784e757eeb366c59ce359d882bf450f867cb07231"}
Nov 01 03:57:59.070319 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:59.070165    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b" event={"ID":"22ab814f-40d0-4ef3-bf2d-0c8029341145","Type":"ContainerStarted","Data":"803fea82170c09e95831353e10feded1c614c9ca11c65bca7ac36f3c465050a6"}
Nov 01 03:57:59.179235 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:57:59.179173    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b" podStartSLOduration=2.179159465 podStartE2EDuration="2.179159465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:57:59.178485874 +0000 UTC m=+947.740518890" watchObservedRunningTime="2026-11-01 03:57:59.179159465 +0000 UTC m=+947.741192482"
Nov 01 03:58:00.077730 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:00.077686    4882 generic.go:356] "Generic (PLEG): container finished" podID="22ab814f-40d0-4ef3-bf2d-0c8029341145" containerID="afede98c939c3d38671842c784e757eeb366c59ce359d882bf450f867cb07231" exitCode=0
Nov 01 03:58:00.078105 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:00.077768    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b" event={"ID":"22ab814f-40d0-4ef3-bf2d-0c8029341145","Type":"ContainerDied","Data":"afede98c939c3d38671842c784e757eeb366c59ce359d882bf450f867cb07231"}
Nov 01 03:58:01.220103 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:01.220076    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:58:01.383407 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:01.383323    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-openshift-ca\") pod \"22ab814f-40d0-4ef3-bf2d-0c8029341145\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") "
Nov 01 03:58:01.383582 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:01.383471    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ab814f-40d0-4ef3-bf2d-0c8029341145-kube-api-access-g7t9h\" (UniqueName: \"kubernetes.io/projected/22ab814f-40d0-4ef3-bf2d-0c8029341145-kube-api-access-g7t9h\") pod \"22ab814f-40d0-4ef3-bf2d-0c8029341145\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") "
Nov 01 03:58:01.383582 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:01.383570    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-xks-ca\") pod \"22ab814f-40d0-4ef3-bf2d-0c8029341145\" (UID: \"22ab814f-40d0-4ef3-bf2d-0c8029341145\") "
Nov 01 03:58:01.383800 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:01.383776    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-openshift-ca" pod "22ab814f-40d0-4ef3-bf2d-0c8029341145" (UID: "22ab814f-40d0-4ef3-bf2d-0c8029341145"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:58:01.383942 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:01.383922    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:01.385561 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:01.385539    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-xks-ca" pod "22ab814f-40d0-4ef3-bf2d-0c8029341145" (UID: "22ab814f-40d0-4ef3-bf2d-0c8029341145"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:58:01.385679 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:01.385659    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ab814f-40d0-4ef3-bf2d-0c8029341145-kube-api-access-g7t9h" pod "22ab814f-40d0-4ef3-bf2d-0c8029341145" (UID: "22ab814f-40d0-4ef3-bf2d-0c8029341145"). InnerVolumeSpecName "kube-api-access-g7t9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:58:01.492623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:01.485342    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7t9h\" (UniqueName: \"kubernetes.io/projected/22ab814f-40d0-4ef3-bf2d-0c8029341145-kube-api-access-g7t9h\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:01.492623 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:01.485829    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ab814f-40d0-4ef3-bf2d-0c8029341145-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:02.091898 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:02.091870    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"
Nov 01 03:58:02.092072 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:02.091865    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b" event={"ID":"22ab814f-40d0-4ef3-bf2d-0c8029341145","Type":"ContainerDied","Data":"803fea82170c09e95831353e10feded1c614c9ca11c65bca7ac36f3c465050a6"}
Nov 01 03:58:02.092072 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:02.091999    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="803fea82170c09e95831353e10feded1c614c9ca11c65bca7ac36f3c465050a6"
Nov 01 03:58:10.020228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.020193    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"]
Nov 01 03:58:10.020709 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.020695    4882 cpu_manager.go:414] "containerMap: removing container" podUID="22ab814f-40d0-4ef3-bf2d-0c8029341145" containerName="revoke-keys"
Nov 01 03:58:10.020753 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.020710    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ab814f-40d0-4ef3-bf2d-0c8029341145" containerName="revoke-keys"
Nov 01 03:58:10.020802 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.020793    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ab814f-40d0-4ef3-bf2d-0c8029341145" containerName="revoke-keys"
Nov 01 03:58:10.023978 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.023958    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:10.074710 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.074687    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:58:10.077909 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.077880    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bk59\" (UniqueName: \"kubernetes.io/projected/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-kube-api-access-5bk59\") pod \"maas-api-revoke-sub-7b0ba4d5-vlpq4\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:10.078053 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.078023    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0ba4d5-vlpq4\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:10.078139 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.078116    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0ba4d5-vlpq4\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:10.121047 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.121023    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 03:58:10.126146 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.126112    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"]
Nov 01 03:58:10.178868 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.178841    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0ba4d5-vlpq4\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:10.179017 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.178873    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0ba4d5-vlpq4\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:10.179017 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.178939    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bk59\" (UniqueName: \"kubernetes.io/projected/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-kube-api-access-5bk59\") pod \"maas-api-revoke-sub-7b0ba4d5-vlpq4\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:10.179625 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.179601    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0ba4d5-vlpq4\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:10.181176 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.181155    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0ba4d5-vlpq4\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:10.240660 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.240640    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bk59\" (UniqueName: \"kubernetes.io/projected/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-kube-api-access-5bk59\") pod \"maas-api-revoke-sub-7b0ba4d5-vlpq4\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:10.336934 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.336852    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:10.705201 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:10.705177    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"]
Nov 01 03:58:10.707662 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:58:10.707631    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a35bd3_47bf_4bed_b5c9_7d34bc77d377.slice/crio-6fc7b4971b1e8f85d5de9f161d8b4a502ac6d21b9cd31ad1e93419569ea7dba8 WatchSource:0}: Error finding container 6fc7b4971b1e8f85d5de9f161d8b4a502ac6d21b9cd31ad1e93419569ea7dba8: Status 404 returned error can't find the container with id 6fc7b4971b1e8f85d5de9f161d8b4a502ac6d21b9cd31ad1e93419569ea7dba8
Nov 01 03:58:11.148329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:11.148293    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4" event={"ID":"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377","Type":"ContainerStarted","Data":"2c16c45267878feecc26b2dac7f3385fb8f6547bda790ef22c5e1d9d54191975"}
Nov 01 03:58:11.148735 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:11.148355    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4" event={"ID":"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377","Type":"ContainerStarted","Data":"6fc7b4971b1e8f85d5de9f161d8b4a502ac6d21b9cd31ad1e93419569ea7dba8"}
Nov 01 03:58:11.257634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:11.257545    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4" podStartSLOduration=2.257531545 podStartE2EDuration="2.257531545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:11.255567234 +0000 UTC m=+959.817600250" watchObservedRunningTime="2026-11-01 03:58:11.257531545 +0000 UTC m=+959.819564558"
Nov 01 03:58:12.156079 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:12.155976    4882 generic.go:356] "Generic (PLEG): container finished" podID="d8a35bd3-47bf-4bed-b5c9-7d34bc77d377" containerID="2c16c45267878feecc26b2dac7f3385fb8f6547bda790ef22c5e1d9d54191975" exitCode=0
Nov 01 03:58:12.156079 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:12.156069    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4" event={"ID":"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377","Type":"ContainerDied","Data":"2c16c45267878feecc26b2dac7f3385fb8f6547bda790ef22c5e1d9d54191975"}
Nov 01 03:58:13.307964 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:13.307932    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:13.425839 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:13.425755    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-xks-ca\") pod \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") "
Nov 01 03:58:13.426011 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:13.425861    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-openshift-ca\") pod \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") "
Nov 01 03:58:13.426011 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:13.426005    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-kube-api-access-5bk59\" (UniqueName: \"kubernetes.io/projected/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-kube-api-access-5bk59\") pod \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\" (UID: \"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377\") "
Nov 01 03:58:13.426270 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:13.426236    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-openshift-ca" pod "d8a35bd3-47bf-4bed-b5c9-7d34bc77d377" (UID: "d8a35bd3-47bf-4bed-b5c9-7d34bc77d377"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:58:13.426421 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:13.426402    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:13.428020 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:13.427990    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-xks-ca" pod "d8a35bd3-47bf-4bed-b5c9-7d34bc77d377" (UID: "d8a35bd3-47bf-4bed-b5c9-7d34bc77d377"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:58:13.428272 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:13.428250    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-kube-api-access-5bk59" pod "d8a35bd3-47bf-4bed-b5c9-7d34bc77d377" (UID: "d8a35bd3-47bf-4bed-b5c9-7d34bc77d377"). InnerVolumeSpecName "kube-api-access-5bk59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:58:13.527865 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:13.527838    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:13.527865 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:13.527865    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bk59\" (UniqueName: \"kubernetes.io/projected/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377-kube-api-access-5bk59\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:14.168754 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:14.168722    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"
Nov 01 03:58:14.168754 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:14.168734    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4" event={"ID":"d8a35bd3-47bf-4bed-b5c9-7d34bc77d377","Type":"ContainerDied","Data":"6fc7b4971b1e8f85d5de9f161d8b4a502ac6d21b9cd31ad1e93419569ea7dba8"}
Nov 01 03:58:14.169003 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:14.168802    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fc7b4971b1e8f85d5de9f161d8b4a502ac6d21b9cd31ad1e93419569ea7dba8"
Nov 01 03:58:18.303102 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.303061    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"]
Nov 01 03:58:18.303888 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.303865    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d8a35bd3-47bf-4bed-b5c9-7d34bc77d377" containerName="revoke-keys"
Nov 01 03:58:18.303946 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.303892    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a35bd3-47bf-4bed-b5c9-7d34bc77d377" containerName="revoke-keys"
Nov 01 03:58:18.304058 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.304045    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a35bd3-47bf-4bed-b5c9-7d34bc77d377" containerName="revoke-keys"
Nov 01 03:58:18.308993 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.308973    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:18.359481 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.359424    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"]
Nov 01 03:58:18.388248 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.388210    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96fvm\" (UniqueName: \"kubernetes.io/projected/1d3e197f-148e-4fa0-a54e-081112abbebe-kube-api-access-96fvm\") pod \"maas-api-revoke-sub-d2bb3559-ps8dr\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:18.388456 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.388397    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bb3559-ps8dr\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:18.388555 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.388484    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bb3559-ps8dr\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:18.402408 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.402372    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:58:18.402564 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.402526    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 03:58:18.489813 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.489781    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bb3559-ps8dr\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:18.490013 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.489842    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bb3559-ps8dr\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:18.490013 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.489965    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96fvm\" (UniqueName: \"kubernetes.io/projected/1d3e197f-148e-4fa0-a54e-081112abbebe-kube-api-access-96fvm\") pod \"maas-api-revoke-sub-d2bb3559-ps8dr\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:18.490680 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.490651    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bb3559-ps8dr\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:18.492404 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.492378    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bb3559-ps8dr\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:18.547298 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.547262    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96fvm\" (UniqueName: \"kubernetes.io/projected/1d3e197f-148e-4fa0-a54e-081112abbebe-kube-api-access-96fvm\") pod \"maas-api-revoke-sub-d2bb3559-ps8dr\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:18.627493 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:18.627408    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:19.002246 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:19.002203    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"]
Nov 01 03:58:19.005672 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:58:19.005638    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3e197f_148e_4fa0_a54e_081112abbebe.slice/crio-849b33a3eda8639cc51085fb98f89b0f554b4e7ff5b3a069cbc58ffcbe4ad424 WatchSource:0}: Error finding container 849b33a3eda8639cc51085fb98f89b0f554b4e7ff5b3a069cbc58ffcbe4ad424: Status 404 returned error can't find the container with id 849b33a3eda8639cc51085fb98f89b0f554b4e7ff5b3a069cbc58ffcbe4ad424
Nov 01 03:58:19.201451 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:19.201411    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr" event={"ID":"1d3e197f-148e-4fa0-a54e-081112abbebe","Type":"ContainerStarted","Data":"849b33a3eda8639cc51085fb98f89b0f554b4e7ff5b3a069cbc58ffcbe4ad424"}
Nov 01 03:58:20.211601 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:20.211563    4882 generic.go:356] "Generic (PLEG): container finished" podID="1d3e197f-148e-4fa0-a54e-081112abbebe" containerID="6a55a87925dcbae7d028b695c7e7cfac6115b90fa890f0ce481bd8f332afd316" exitCode=0
Nov 01 03:58:20.212094 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:20.211619    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr" event={"ID":"1d3e197f-148e-4fa0-a54e-081112abbebe","Type":"ContainerDied","Data":"6a55a87925dcbae7d028b695c7e7cfac6115b90fa890f0ce481bd8f332afd316"}
Nov 01 03:58:21.359845 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:21.359819    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:21.520735 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:21.520657    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-openshift-ca\") pod \"1d3e197f-148e-4fa0-a54e-081112abbebe\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") "
Nov 01 03:58:21.520886 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:21.520805    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3e197f-148e-4fa0-a54e-081112abbebe-kube-api-access-96fvm\" (UniqueName: \"kubernetes.io/projected/1d3e197f-148e-4fa0-a54e-081112abbebe-kube-api-access-96fvm\") pod \"1d3e197f-148e-4fa0-a54e-081112abbebe\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") "
Nov 01 03:58:21.520927 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:21.520916    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-xks-ca\") pod \"1d3e197f-148e-4fa0-a54e-081112abbebe\" (UID: \"1d3e197f-148e-4fa0-a54e-081112abbebe\") "
Nov 01 03:58:21.521006 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:21.520971    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-openshift-ca" pod "1d3e197f-148e-4fa0-a54e-081112abbebe" (UID: "1d3e197f-148e-4fa0-a54e-081112abbebe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:58:21.521293 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:21.521272    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:21.522906 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:21.522881    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3e197f-148e-4fa0-a54e-081112abbebe-kube-api-access-96fvm" pod "1d3e197f-148e-4fa0-a54e-081112abbebe" (UID: "1d3e197f-148e-4fa0-a54e-081112abbebe"). InnerVolumeSpecName "kube-api-access-96fvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:58:21.522974 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:21.522883    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-xks-ca" pod "1d3e197f-148e-4fa0-a54e-081112abbebe" (UID: "1d3e197f-148e-4fa0-a54e-081112abbebe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:58:21.622645 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:21.622615    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96fvm\" (UniqueName: \"kubernetes.io/projected/1d3e197f-148e-4fa0-a54e-081112abbebe-kube-api-access-96fvm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:21.622645 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:21.622642    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d3e197f-148e-4fa0-a54e-081112abbebe-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:22.225837 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:22.225805    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"
Nov 01 03:58:22.226001 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:22.225806    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr" event={"ID":"1d3e197f-148e-4fa0-a54e-081112abbebe","Type":"ContainerDied","Data":"849b33a3eda8639cc51085fb98f89b0f554b4e7ff5b3a069cbc58ffcbe4ad424"}
Nov 01 03:58:22.226001 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:22.225931    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="849b33a3eda8639cc51085fb98f89b0f554b4e7ff5b3a069cbc58ffcbe4ad424"
Nov 01 03:58:26.146953 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.146920    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"]
Nov 01 03:58:26.147384 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.147371    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1d3e197f-148e-4fa0-a54e-081112abbebe" containerName="revoke-keys"
Nov 01 03:58:26.147457 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.147385    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3e197f-148e-4fa0-a54e-081112abbebe" containerName="revoke-keys"
Nov 01 03:58:26.147502 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.147481    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3e197f-148e-4fa0-a54e-081112abbebe" containerName="revoke-keys"
Nov 01 03:58:26.150677 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.150657    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.245191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.245158    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"e2e-conflict-a-9e9797da-kserve-self-signed-certs\""
Nov 01 03:58:26.248297 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.248271    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"]
Nov 01 03:58:26.273556 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.273525    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-home\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.273827 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.273807    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-model-cache\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.273970 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.273955    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-dshm\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.274084 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.274069    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfvct\" (UniqueName: \"kubernetes.io/projected/03db033d-d120-4ae6-8610-b2ebad7510dd-kube-api-access-gfvct\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.274279 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.274242    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-tmp-dir\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.274387 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.274326    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03db033d-d120-4ae6-8610-b2ebad7510dd-tls-certs\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.375403 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.375367    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-home\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.375593 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.375499    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-model-cache\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.375593 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.375531    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-dshm\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.375593 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.375552    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfvct\" (UniqueName: \"kubernetes.io/projected/03db033d-d120-4ae6-8610-b2ebad7510dd-kube-api-access-gfvct\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.375775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.375697    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-tmp-dir\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.375775 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.375735    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03db033d-d120-4ae6-8610-b2ebad7510dd-tls-certs\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.375882 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.375797    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-home\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.375933 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.375879    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-model-cache\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.376077 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.376056    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-tmp-dir\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.377767 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.377747    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-dshm\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.378198 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.378182    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03db033d-d120-4ae6-8610-b2ebad7510dd-tls-certs\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.432318 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.432260    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfvct\" (UniqueName: \"kubernetes.io/projected/03db033d-d120-4ae6-8610-b2ebad7510dd-kube-api-access-gfvct\") pod \"e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.462832 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.462797    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:26.851230 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:26.851203    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"]
Nov 01 03:58:26.853220 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:58:26.853182    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03db033d_d120_4ae6_8610_b2ebad7510dd.slice/crio-425dfc3fdc7d9dcc5994d6ab861b838b4783cb140a2f6aed21003c00d568f90e WatchSource:0}: Error finding container 425dfc3fdc7d9dcc5994d6ab861b838b4783cb140a2f6aed21003c00d568f90e: Status 404 returned error can't find the container with id 425dfc3fdc7d9dcc5994d6ab861b838b4783cb140a2f6aed21003c00d568f90e
Nov 01 03:58:27.257211 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:27.257177    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz" event={"ID":"03db033d-d120-4ae6-8610-b2ebad7510dd","Type":"ContainerStarted","Data":"cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6"}
Nov 01 03:58:27.257211 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:27.257217    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz" event={"ID":"03db033d-d120-4ae6-8610-b2ebad7510dd","Type":"ContainerStarted","Data":"425dfc3fdc7d9dcc5994d6ab861b838b4783cb140a2f6aed21003c00d568f90e"}
Nov 01 03:58:27.366821 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:27.366771    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz" podStartSLOduration=1.366758434 podStartE2EDuration="1.366758434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:27.365684229 +0000 UTC m=+975.927717247" watchObservedRunningTime="2026-11-01 03:58:27.366758434 +0000 UTC m=+975.928791448"
Nov 01 03:58:27.463874 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:27.463843    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:28.722727 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:58:28.722692    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:58:29.072927 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.072844    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"]
Nov 01 03:58:29.076769 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.076749    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:29.104195 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.104163    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9751f1-mq6vt\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:29.104313 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.104292    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hl4q\" (UniqueName: \"kubernetes.io/projected/e69778b9-3a55-4747-b34a-056c75dc57f5-kube-api-access-2hl4q\") pod \"maas-api-revoke-sub-2a9751f1-mq6vt\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:29.104404 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.104391    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9751f1-mq6vt\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:29.131151 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.131125    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"]
Nov 01 03:58:29.173376 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.173350    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:58:29.173508 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.173477    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 03:58:29.205042 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.205017    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9751f1-mq6vt\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:29.205205 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.205077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hl4q\" (UniqueName: \"kubernetes.io/projected/e69778b9-3a55-4747-b34a-056c75dc57f5-kube-api-access-2hl4q\") pod \"maas-api-revoke-sub-2a9751f1-mq6vt\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:29.205205 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.205141    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9751f1-mq6vt\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:29.205851 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.205823    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9751f1-mq6vt\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:29.207483 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.207451    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9751f1-mq6vt\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:29.264228 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.264205    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hl4q\" (UniqueName: \"kubernetes.io/projected/e69778b9-3a55-4747-b34a-056c75dc57f5-kube-api-access-2hl4q\") pod \"maas-api-revoke-sub-2a9751f1-mq6vt\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:29.392763 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.392696    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:29.760575 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:29.760547    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"]
Nov 01 03:58:29.763061 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:58:29.763034    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69778b9_3a55_4747_b34a_056c75dc57f5.slice/crio-5702bb5b501941e66ac80db14b5027b69ca8b1e0339357b167603f8c71189c41 WatchSource:0}: Error finding container 5702bb5b501941e66ac80db14b5027b69ca8b1e0339357b167603f8c71189c41: Status 404 returned error can't find the container with id 5702bb5b501941e66ac80db14b5027b69ca8b1e0339357b167603f8c71189c41
Nov 01 03:58:30.275882 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:30.275845    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt" event={"ID":"e69778b9-3a55-4747-b34a-056c75dc57f5","Type":"ContainerStarted","Data":"757246b3f9367ae65745de0eb897c8bea3eaa7164f6735af802901b58908bd87"}
Nov 01 03:58:30.276048 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:30.275939    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt" event={"ID":"e69778b9-3a55-4747-b34a-056c75dc57f5","Type":"ContainerStarted","Data":"5702bb5b501941e66ac80db14b5027b69ca8b1e0339357b167603f8c71189c41"}
Nov 01 03:58:30.385743 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:30.385644    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt" podStartSLOduration=1.385629296 podStartE2EDuration="1.385629296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:30.382943473 +0000 UTC m=+978.944976472" watchObservedRunningTime="2026-11-01 03:58:30.385629296 +0000 UTC m=+978.947662314"
Nov 01 03:58:31.283369 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.283334    4882 generic.go:356] "Generic (PLEG): container finished" podID="e69778b9-3a55-4747-b34a-056c75dc57f5" containerID="757246b3f9367ae65745de0eb897c8bea3eaa7164f6735af802901b58908bd87" exitCode=0
Nov 01 03:58:31.283369 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.283372    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt" event={"ID":"e69778b9-3a55-4747-b34a-056c75dc57f5","Type":"ContainerDied","Data":"757246b3f9367ae65745de0eb897c8bea3eaa7164f6735af802901b58908bd87"}
Nov 01 03:58:31.382618 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.382584    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"]
Nov 01 03:58:31.386323 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.386306    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.428038 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.428011    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.428185 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.428041    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.428185 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.428150    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.428264 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.428205    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6df42f-9caf-4a86-a91a-2a51461a8645-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.428329 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.428313    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.428369 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.428356    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.428404 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.428378    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.428703 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.428671    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrhwt\" (UniqueName: \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-kube-api-access-hrhwt\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.428807 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.428729    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.435009 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.434986    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a9c5b3-openshift-default-dockercfg-88kq5\""
Nov 01 03:58:31.471229 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.471202    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"]
Nov 01 03:58:31.529859 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.529816    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530007 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.529889    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6df42f-9caf-4a86-a91a-2a51461a8645-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530007 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.529929    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530007 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.529958    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530007 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.529977    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530259 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.530026    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrhwt\" (UniqueName: \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-kube-api-access-hrhwt\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530259 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.530051    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530259 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.530104    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530259 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.530127    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530445 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.530257    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530445 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.530316    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530521 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.530486    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530743 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.530717    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6df42f-9caf-4a86-a91a-2a51461a8645-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.530854 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.530743    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.532494 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.532467    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.532669 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.532650    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.587134 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.587064    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.587304 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.587282    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrhwt\" (UniqueName: \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-kube-api-access-hrhwt\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:31.700081 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:31.700053    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:32.122366 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.122337    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"]
Nov 01 03:58:32.124033 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:58:32.124004    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6df42f_9caf_4a86_a91a_2a51461a8645.slice/crio-ad765f569285779c2b055a5ff1ce48624812298615aa795dd49552d820f867b8 WatchSource:0}: Error finding container ad765f569285779c2b055a5ff1ce48624812298615aa795dd49552d820f867b8: Status 404 returned error can't find the container with id ad765f569285779c2b055a5ff1ce48624812298615aa795dd49552d820f867b8
Nov 01 03:58:32.126659 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.126629    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:32.126739 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.126703    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:32.126791 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.126743    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:32.291679 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.291629    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" event={"ID":"8e6df42f-9caf-4a86-a91a-2a51461a8645","Type":"ContainerStarted","Data":"4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55"}
Nov 01 03:58:32.292334 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.291713    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" event={"ID":"8e6df42f-9caf-4a86-a91a-2a51461a8645","Type":"ContainerStarted","Data":"ad765f569285779c2b055a5ff1ce48624812298615aa795dd49552d820f867b8"}
Nov 01 03:58:32.405664 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.405603    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" podStartSLOduration=1.405583525 podStartE2EDuration="1.405583525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:32.404798606 +0000 UTC m=+980.966831622" watchObservedRunningTime="2026-11-01 03:58:32.405583525 +0000 UTC m=+980.967616543"
Nov 01 03:58:32.413628 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.413603    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:32.439734 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.439700    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69778b9-3a55-4747-b34a-056c75dc57f5-kube-api-access-2hl4q\" (UniqueName: \"kubernetes.io/projected/e69778b9-3a55-4747-b34a-056c75dc57f5-kube-api-access-2hl4q\") pod \"e69778b9-3a55-4747-b34a-056c75dc57f5\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") "
Nov 01 03:58:32.439887 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.439805    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-openshift-ca\") pod \"e69778b9-3a55-4747-b34a-056c75dc57f5\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") "
Nov 01 03:58:32.439887 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.439838    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-xks-ca\") pod \"e69778b9-3a55-4747-b34a-056c75dc57f5\" (UID: \"e69778b9-3a55-4747-b34a-056c75dc57f5\") "
Nov 01 03:58:32.440237 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.440207    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-openshift-ca" pod "e69778b9-3a55-4747-b34a-056c75dc57f5" (UID: "e69778b9-3a55-4747-b34a-056c75dc57f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:58:32.441870 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.441846    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69778b9-3a55-4747-b34a-056c75dc57f5-kube-api-access-2hl4q" pod "e69778b9-3a55-4747-b34a-056c75dc57f5" (UID: "e69778b9-3a55-4747-b34a-056c75dc57f5"). InnerVolumeSpecName "kube-api-access-2hl4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:58:32.441960 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.441879    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-xks-ca" pod "e69778b9-3a55-4747-b34a-056c75dc57f5" (UID: "e69778b9-3a55-4747-b34a-056c75dc57f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:58:32.544482 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.541172    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hl4q\" (UniqueName: \"kubernetes.io/projected/e69778b9-3a55-4747-b34a-056c75dc57f5-kube-api-access-2hl4q\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:32.544482 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.541208    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:32.544482 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.541219    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69778b9-3a55-4747-b34a-056c75dc57f5-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 03:58:32.701094 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.701065    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:32.807411 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.807353    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"]
Nov 01 03:58:32.807911 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.807894    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e69778b9-3a55-4747-b34a-056c75dc57f5" containerName="revoke-keys"
Nov 01 03:58:32.807911 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.807911    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69778b9-3a55-4747-b34a-056c75dc57f5" containerName="revoke-keys"
Nov 01 03:58:32.808039 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.807990    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69778b9-3a55-4747-b34a-056c75dc57f5" containerName="revoke-keys"
Nov 01 03:58:32.811584 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.811562    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.844702 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.844676    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.844835 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.844747    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.844911 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.844830    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.844911 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.844883    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.845021 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.844930    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.845021 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.844969    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6hq9\" (UniqueName: \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-kube-api-access-v6hq9\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.845141 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.845126    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.845193 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.845158    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d36b171-563c-4230-b2fc-aa810dceaa0b-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.845247 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.845191    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.870230 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.870201    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"]
Nov 01 03:58:32.946479 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.946405    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.946668 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.946504    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.946668 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.946553    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.946668 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.946601    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.946668 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.946639    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6hq9\" (UniqueName: \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-kube-api-access-v6hq9\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.946885 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.946683    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.946885 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.946720    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d36b171-563c-4230-b2fc-aa810dceaa0b-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.946885 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.946770    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.946885 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.946851    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.946885 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.946874    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.947113 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.947020    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.947266 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.947247    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.947347 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.947289    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.947584 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.947561    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d36b171-563c-4230-b2fc-aa810dceaa0b-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.949020 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.948997    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:32.949257 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:32.949235    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:33.003877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.003809    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:33.003877 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.003820    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6hq9\" (UniqueName: \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-kube-api-access-v6hq9\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:33.124958 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.124918    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:33.301549 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.301466    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"
Nov 01 03:58:33.301963 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.301462    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt" event={"ID":"e69778b9-3a55-4747-b34a-056c75dc57f5","Type":"ContainerDied","Data":"5702bb5b501941e66ac80db14b5027b69ca8b1e0339357b167603f8c71189c41"}
Nov 01 03:58:33.301963 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.301598    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5702bb5b501941e66ac80db14b5027b69ca8b1e0339357b167603f8c71189c41"
Nov 01 03:58:33.597262 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.597213    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:33.597358 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.597294    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:33.597358 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.597333    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:33.607263 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.607237    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"]
Nov 01 03:58:33.702051 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.702017    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:58:33.702191 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:33.702076    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 03:58:34.318560 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:34.318510    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" event={"ID":"4d36b171-563c-4230-b2fc-aa810dceaa0b","Type":"ContainerStarted","Data":"d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089"}
Nov 01 03:58:34.319622 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:34.319589    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" event={"ID":"4d36b171-563c-4230-b2fc-aa810dceaa0b","Type":"ContainerStarted","Data":"9642449ca731899256ddf5d9ad15be8426ea37853e819910337e9fbc53a91315"}
Nov 01 03:58:34.703018 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:34.702980    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:58:34.703282 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:34.703246    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 03:58:35.127608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:35.127016    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:35.127608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:35.127206    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:58:35.127608 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:35.127251    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:58:35.702830 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:35.702792    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:58:35.703422 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:35.703389    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 03:58:36.126519 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:36.125858    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:58:36.126519 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:36.125927    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:58:36.470453 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:36.469443    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:36.545083 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:36.545053    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:36.661383 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:36.661316    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" podStartSLOduration=4.661296501 podStartE2EDuration="4.661296501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:34.48501828 +0000 UTC m=+983.047051309" watchObservedRunningTime="2026-11-01 03:58:36.661296501 +0000 UTC m=+985.223329521"
Nov 01 03:58:36.706451 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:36.704839    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:58:36.706451 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:36.704908    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 03:58:37.128511 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:37.126911    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:58:37.128511 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:37.126994    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:58:37.357955 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:37.357926    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 03:58:37.704464 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:37.703494    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:58:37.704464 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:37.703553    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 03:58:38.125639 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:38.125567    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:58:38.126086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:38.126061    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:58:38.700628 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:38.700598    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:58:38.700787 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:38.700647    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 03:58:39.128513 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:39.127797    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:58:39.128513 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:39.127878    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:58:40.128888 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.128846    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 01 03:58:40.129384 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.129075    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 01 03:58:40.132687 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.132659    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"]
Nov 01 03:58:40.136874 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.136849    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.234828 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.234798    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"]
Nov 01 03:58:40.242139 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.242102    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.242254 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.242162    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.242314 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.242285    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.242501 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.242481    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.242557 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.242542    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.242599 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.242582    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgnj\" (UniqueName: \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-kube-api-access-4sgnj\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.242706 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.242686    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.242767 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.242737    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.242803 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.242770    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.343999 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.343966    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.343999 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344005    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgnj\" (UniqueName: \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-kube-api-access-4sgnj\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.344221 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344044    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.344221 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.344221 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344104    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.344221 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344145    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.344221 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344186    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.344221 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344213    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.344567 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344339    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.344634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344577    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.344634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344609    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.344742 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344640    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.344919 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.344894    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.345259 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.345235    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.347309 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.347277    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.347714 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.347686    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.363808 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.360115    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:40.363808 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.361002    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:40.363808 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.363032    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 03:58:40.401521 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.401443    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgnj\" (UniqueName: \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-kube-api-access-4sgnj\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.401521 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.401477    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-jxfk6\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.451145 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.451110    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:40.873141 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.873113    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"]
Nov 01 03:58:40.875867 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:58:40.875838    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b33ad10_74bb_4488_bda7_a04a1df5cc8d.slice/crio-7235d3fc3999bda56ec0b1174d8b0920093d5aab609f4e272edb6e21c4cf95e3 WatchSource:0}: Error finding container 7235d3fc3999bda56ec0b1174d8b0920093d5aab609f4e272edb6e21c4cf95e3: Status 404 returned error can't find the container with id 7235d3fc3999bda56ec0b1174d8b0920093d5aab609f4e272edb6e21c4cf95e3
Nov 01 03:58:40.878974 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.878943    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:40.879078 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.879007    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:40.879078 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:40.879045    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:41.131172 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:41.131098    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:41.131638 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:41.131482    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:41.132284 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:41.132265    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 03:58:41.381783 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:41.381679    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" event={"ID":"6b33ad10-74bb-4488-bda7-a04a1df5cc8d","Type":"ContainerStarted","Data":"8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b"}
Nov 01 03:58:41.381783 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:41.381742    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" event={"ID":"6b33ad10-74bb-4488-bda7-a04a1df5cc8d","Type":"ContainerStarted","Data":"7235d3fc3999bda56ec0b1174d8b0920093d5aab609f4e272edb6e21c4cf95e3"}
Nov 01 03:58:41.452044 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:41.452015    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:41.453768 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:41.453743    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jxfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:58:41.453903 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:41.453792    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:58:41.495713 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:41.495655    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podStartSLOduration=1.495636764 podStartE2EDuration="1.495636764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:41.495274176 +0000 UTC m=+990.057307193" watchObservedRunningTime="2026-11-01 03:58:41.495636764 +0000 UTC m=+990.057669783"
Nov 01 03:58:42.452067 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:42.452029    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jxfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:58:42.452493 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:42.452084    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:58:43.452025 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:43.451990    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jxfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:58:43.452221 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:43.452053    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:58:44.451539 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:44.451504    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jxfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:58:44.451698 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:44.451556    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:58:45.451838 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:45.451803    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jxfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:58:45.452321 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:45.451860    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:58:46.452269 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:46.452231    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jxfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:58:46.452694 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:46.452293    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:58:47.451678 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:47.451636    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jxfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:58:47.451885 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:47.451722    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:58:48.452101 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:48.452065    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jxfk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 03:58:48.452511 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:48.452140    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 03:58:49.455653 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:49.455622    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:49.456062 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:49.456040    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:49.456696 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:49.456676    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 03:58:55.234706 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.234667    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"]
Nov 01 03:58:55.243010 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.242979    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.255196 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.255167    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.255309 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.255238    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.255309 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.255280    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwgdz\" (UniqueName: \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-kube-api-access-fwgdz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.255471 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.255315    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.255471 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.255403    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.255471 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.255466    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.255632 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.255484    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.255632 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.255601    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.255735 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.255639    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c0d040-6d16-4890-9595-43e09ef4f0b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.331550 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.331522    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"]
Nov 01 03:58:55.356621 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.356590    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.356739 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.356643    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c0d040-6d16-4890-9595-43e09ef4f0b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.356739 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.356716    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.356873 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.356850    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.357031 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.357007    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwgdz\" (UniqueName: \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-kube-api-access-fwgdz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.357031 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.357016    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.357179 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.357136    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.357237 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.357188    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.357237 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.357231    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.357349 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.357255    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.357449 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.357413    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c0d040-6d16-4890-9595-43e09ef4f0b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.357634 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.357607    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.357801 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.357773    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.357978 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.357957    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.359087 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.359067    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.359435 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.359408    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.414485 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.414459    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.414675 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.414603    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwgdz\" (UniqueName: \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-kube-api-access-fwgdz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-flvv6\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.557615 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.557527    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:55.982327 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.982298    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"]
Nov 01 03:58:55.983722 ip-10-0-129-252 kubenswrapper[4882]: W1101 03:58:55.983696    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c0d040_6d16_4890_9595_43e09ef4f0b0.slice/crio-cf903601997c0b54b462092e1767b72ee66ae48f290f35c2f0e2d78506a00918 WatchSource:0}: Error finding container cf903601997c0b54b462092e1767b72ee66ae48f290f35c2f0e2d78506a00918: Status 404 returned error can't find the container with id cf903601997c0b54b462092e1767b72ee66ae48f290f35c2f0e2d78506a00918
Nov 01 03:58:55.986046 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.986015    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:55.986144 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.986096    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:55.986144 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:55.986128    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:58:56.492102 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:56.490958    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" event={"ID":"86c0d040-6d16-4890-9595-43e09ef4f0b0","Type":"ContainerStarted","Data":"9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434"}
Nov 01 03:58:56.492102 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:56.491029    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" event={"ID":"86c0d040-6d16-4890-9595-43e09ef4f0b0","Type":"ContainerStarted","Data":"cf903601997c0b54b462092e1767b72ee66ae48f290f35c2f0e2d78506a00918"}
Nov 01 03:58:56.558398 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:56.558368    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:58:56.606150 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:56.606098    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" podStartSLOduration=1.6060743720000001 podStartE2EDuration="1.606074372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:56.60395114 +0000 UTC m=+1005.165984156" watchObservedRunningTime="2026-11-01 03:58:56.606074372 +0000 UTC m=+1005.168107388"
Nov 01 03:58:57.559664 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:57.559633    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:58:57.560074 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:57.559684    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:58:58.558492 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:58.558456    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:58:58.558661 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:58.558515    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:58:58.723349 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:58:58.723313    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:58:59.558386 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:59.558349    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:58:59.558580 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:58:59.558406    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:59:00.558417 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:59:00.558386    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:59:00.558837 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:59:00.558465    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:59:01.558698 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:59:01.558661    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:59:01.559086 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:59:01.558714    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:59:02.558018 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:59:02.557982    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 03:59:02.558189 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:59:02.558037    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 03:59:04.418858 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:59:04.418825    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:59:04.419341 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:59:04.419234    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:59:04.419979 ip-10-0-129-252 kubenswrapper[4882]: I1101 03:59:04.419959    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 03:59:28.723293 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:59:28.723203    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 03:59:58.723420 ip-10-0-129-252 kubenswrapper[4882]: E1101 03:59:58.723378    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:00:00.171662 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.171627    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"]
Nov 01 04:00:00.175564 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.175536    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"
Nov 01 04:00:00.182363 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.182338    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"]
Nov 01 04:00:00.186526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.186508    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"]
Nov 01 04:00:00.186644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.186622    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"
Nov 01 04:00:00.190868 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.190849    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"
Nov 01 04:00:00.215487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.215457    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2258d228-bef4-47e4-b4ca-d4ca466fedcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68\" (UID: \"2258d228-bef4-47e4-b4ca-d4ca466fedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"
Nov 01 04:00:00.215659 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.215582    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bszsb\" (UniqueName: \"kubernetes.io/projected/2258d228-bef4-47e4-b4ca-d4ca466fedcf-kube-api-access-bszsb\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68\" (UID: \"2258d228-bef4-47e4-b4ca-d4ca466fedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"
Nov 01 04:00:00.218115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.218094    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"]
Nov 01 04:00:00.221855 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.221840    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"
Nov 01 04:00:00.224714 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.224674    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:00:00.268386 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.268004    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"]
Nov 01 04:00:00.272494 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.272465    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"]
Nov 01 04:00:00.273538 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.273513    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"]
Nov 01 04:00:00.316756 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.316721    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6520c6-1e47-43e5-b2fa-11a77f00d501-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6\" (UID: \"dc6520c6-1e47-43e5-b2fa-11a77f00d501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"
Nov 01 04:00:00.316954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.316768    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszsb\" (UniqueName: \"kubernetes.io/projected/2258d228-bef4-47e4-b4ca-d4ca466fedcf-kube-api-access-bszsb\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68\" (UID: \"2258d228-bef4-47e4-b4ca-d4ca466fedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"
Nov 01 04:00:00.317027 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.316966    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hssmg\" (UniqueName: \"kubernetes.io/projected/dc6520c6-1e47-43e5-b2fa-11a77f00d501-kube-api-access-hssmg\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6\" (UID: \"dc6520c6-1e47-43e5-b2fa-11a77f00d501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"
Nov 01 04:00:00.317027 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.316999    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08587f3b-3fdb-4888-b96a-46472d3e537f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5\" (UID: \"08587f3b-3fdb-4888-b96a-46472d3e537f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"
Nov 01 04:00:00.317155 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.317091    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3043c5b-899e-4e4d-b879-3b3eb798cf04-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891760-n9j9x\" (UID: \"b3043c5b-899e-4e4d-b879-3b3eb798cf04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"
Nov 01 04:00:00.317155 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.317136    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2258d228-bef4-47e4-b4ca-d4ca466fedcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68\" (UID: \"2258d228-bef4-47e4-b4ca-d4ca466fedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"
Nov 01 04:00:00.317256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.317155    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlnwq\" (UniqueName: \"kubernetes.io/projected/08587f3b-3fdb-4888-b96a-46472d3e537f-kube-api-access-xlnwq\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5\" (UID: \"08587f3b-3fdb-4888-b96a-46472d3e537f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"
Nov 01 04:00:00.317307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.317280    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rhqv\" (UniqueName: \"kubernetes.io/projected/b3043c5b-899e-4e4d-b879-3b3eb798cf04-kube-api-access-8rhqv\") pod \"maas-api-key-cleanup-29891760-n9j9x\" (UID: \"b3043c5b-899e-4e4d-b879-3b3eb798cf04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"
Nov 01 04:00:00.317831 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.317801    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2258d228-bef4-47e4-b4ca-d4ca466fedcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68\" (UID: \"2258d228-bef4-47e4-b4ca-d4ca466fedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"
Nov 01 04:00:00.331812 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.331782    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"]
Nov 01 04:00:00.374253 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.374217    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bszsb\" (UniqueName: \"kubernetes.io/projected/2258d228-bef4-47e4-b4ca-d4ca466fedcf-kube-api-access-bszsb\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68\" (UID: \"2258d228-bef4-47e4-b4ca-d4ca466fedcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"
Nov 01 04:00:00.418383 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.418341    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rhqv\" (UniqueName: \"kubernetes.io/projected/b3043c5b-899e-4e4d-b879-3b3eb798cf04-kube-api-access-8rhqv\") pod \"maas-api-key-cleanup-29891760-n9j9x\" (UID: \"b3043c5b-899e-4e4d-b879-3b3eb798cf04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"
Nov 01 04:00:00.418588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.418407    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6520c6-1e47-43e5-b2fa-11a77f00d501-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6\" (UID: \"dc6520c6-1e47-43e5-b2fa-11a77f00d501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"
Nov 01 04:00:00.418588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.418582    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hssmg\" (UniqueName: \"kubernetes.io/projected/dc6520c6-1e47-43e5-b2fa-11a77f00d501-kube-api-access-hssmg\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6\" (UID: \"dc6520c6-1e47-43e5-b2fa-11a77f00d501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"
Nov 01 04:00:00.418697 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.418628    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08587f3b-3fdb-4888-b96a-46472d3e537f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5\" (UID: \"08587f3b-3fdb-4888-b96a-46472d3e537f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"
Nov 01 04:00:00.418697 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.418685    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3043c5b-899e-4e4d-b879-3b3eb798cf04-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891760-n9j9x\" (UID: \"b3043c5b-899e-4e4d-b879-3b3eb798cf04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"
Nov 01 04:00:00.418910 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.418887    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlnwq\" (UniqueName: \"kubernetes.io/projected/08587f3b-3fdb-4888-b96a-46472d3e537f-kube-api-access-xlnwq\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5\" (UID: \"08587f3b-3fdb-4888-b96a-46472d3e537f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"
Nov 01 04:00:00.419123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.419103    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6520c6-1e47-43e5-b2fa-11a77f00d501-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6\" (UID: \"dc6520c6-1e47-43e5-b2fa-11a77f00d501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"
Nov 01 04:00:00.419311 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.419293    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3043c5b-899e-4e4d-b879-3b3eb798cf04-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891760-n9j9x\" (UID: \"b3043c5b-899e-4e4d-b879-3b3eb798cf04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"
Nov 01 04:00:00.419392 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.419374    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08587f3b-3fdb-4888-b96a-46472d3e537f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5\" (UID: \"08587f3b-3fdb-4888-b96a-46472d3e537f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"
Nov 01 04:00:00.475961 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.475933    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rhqv\" (UniqueName: \"kubernetes.io/projected/b3043c5b-899e-4e4d-b879-3b3eb798cf04-kube-api-access-8rhqv\") pod \"maas-api-key-cleanup-29891760-n9j9x\" (UID: \"b3043c5b-899e-4e4d-b879-3b3eb798cf04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"
Nov 01 04:00:00.476121 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.476030    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hssmg\" (UniqueName: \"kubernetes.io/projected/dc6520c6-1e47-43e5-b2fa-11a77f00d501-kube-api-access-hssmg\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6\" (UID: \"dc6520c6-1e47-43e5-b2fa-11a77f00d501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"
Nov 01 04:00:00.476167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.476141    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlnwq\" (UniqueName: \"kubernetes.io/projected/08587f3b-3fdb-4888-b96a-46472d3e537f-kube-api-access-xlnwq\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5\" (UID: \"08587f3b-3fdb-4888-b96a-46472d3e537f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"
Nov 01 04:00:00.491094 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.491063    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"
Nov 01 04:00:00.500579 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.500555    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"
Nov 01 04:00:00.509226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.509198    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"
Nov 01 04:00:00.535711 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.535677    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"
Nov 01 04:00:00.871885 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.871851    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"]
Nov 01 04:00:00.873203 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:00:00.873160    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2258d228_bef4_47e4_b4ca_d4ca466fedcf.slice/crio-33b8de09bd9965d5d6c06909bed6a4813ec7ebd26b17674bfca9d48c4e272ca4 WatchSource:0}: Error finding container 33b8de09bd9965d5d6c06909bed6a4813ec7ebd26b17674bfca9d48c4e272ca4: Status 404 returned error can't find the container with id 33b8de09bd9965d5d6c06909bed6a4813ec7ebd26b17674bfca9d48c4e272ca4
Nov 01 04:00:00.918927 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.918896    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"]
Nov 01 04:00:00.921809 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:00:00.921770    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6520c6_1e47_43e5_b2fa_11a77f00d501.slice/crio-83e5fa6045bbb556a50f35eb601915a2ef02e38cfbaa0941955517696b7831dc WatchSource:0}: Error finding container 83e5fa6045bbb556a50f35eb601915a2ef02e38cfbaa0941955517696b7831dc: Status 404 returned error can't find the container with id 83e5fa6045bbb556a50f35eb601915a2ef02e38cfbaa0941955517696b7831dc
Nov 01 04:00:00.924566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.924538    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"]
Nov 01 04:00:00.925583 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:00:00.925557    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3043c5b_899e_4e4d_b879_3b3eb798cf04.slice/crio-70b2b9ed40691c92941b46ce7b4273ac560742da5c73d61a632dc1989617b6fd WatchSource:0}: Error finding container 70b2b9ed40691c92941b46ce7b4273ac560742da5c73d61a632dc1989617b6fd: Status 404 returned error can't find the container with id 70b2b9ed40691c92941b46ce7b4273ac560742da5c73d61a632dc1989617b6fd
Nov 01 04:00:00.942989 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.942962    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x" event={"ID":"b3043c5b-899e-4e4d-b879-3b3eb798cf04","Type":"ContainerStarted","Data":"70b2b9ed40691c92941b46ce7b4273ac560742da5c73d61a632dc1989617b6fd"}
Nov 01 04:00:00.944796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.944763    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" event={"ID":"2258d228-bef4-47e4-b4ca-d4ca466fedcf","Type":"ContainerStarted","Data":"33b8de09bd9965d5d6c06909bed6a4813ec7ebd26b17674bfca9d48c4e272ca4"}
Nov 01 04:00:00.946255 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.946231    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6" event={"ID":"dc6520c6-1e47-43e5-b2fa-11a77f00d501","Type":"ContainerStarted","Data":"83e5fa6045bbb556a50f35eb601915a2ef02e38cfbaa0941955517696b7831dc"}
Nov 01 04:00:00.953856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:00.953832    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"]
Nov 01 04:00:00.956990 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:00:00.956954    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08587f3b_3fdb_4888_b96a_46472d3e537f.slice/crio-2bfb8adfb45f5d4a0fe3942508c544207c8c1c23088ee784091c1041e84da1ea WatchSource:0}: Error finding container 2bfb8adfb45f5d4a0fe3942508c544207c8c1c23088ee784091c1041e84da1ea: Status 404 returned error can't find the container with id 2bfb8adfb45f5d4a0fe3942508c544207c8c1c23088ee784091c1041e84da1ea
Nov 01 04:00:01.955645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:01.955608    4882 generic.go:356] "Generic (PLEG): container finished" podID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerID="b64521247b1dc794822c95e908f149cf424fefc58544347f2428b08564e6960e" exitCode=60
Nov 01 04:00:01.956145 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:01.955699    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6" event={"ID":"dc6520c6-1e47-43e5-b2fa-11a77f00d501","Type":"ContainerDied","Data":"b64521247b1dc794822c95e908f149cf424fefc58544347f2428b08564e6960e"}
Nov 01 04:00:01.956145 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:01.955963    4882 scope.go:122] "RemoveContainer" containerID="b64521247b1dc794822c95e908f149cf424fefc58544347f2428b08564e6960e"
Nov 01 04:00:01.957578 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:01.957551    4882 generic.go:356] "Generic (PLEG): container finished" podID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerID="228c1b7869fc7ffcdd269468a39e4633fd9dd3f2ffb8e22d491e71813a5f1b52" exitCode=60
Nov 01 04:00:01.957652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:01.957630    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x" event={"ID":"b3043c5b-899e-4e4d-b879-3b3eb798cf04","Type":"ContainerDied","Data":"228c1b7869fc7ffcdd269468a39e4633fd9dd3f2ffb8e22d491e71813a5f1b52"}
Nov 01 04:00:01.957849 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:01.957832    4882 scope.go:122] "RemoveContainer" containerID="228c1b7869fc7ffcdd269468a39e4633fd9dd3f2ffb8e22d491e71813a5f1b52"
Nov 01 04:00:01.959525 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:01.959497    4882 generic.go:356] "Generic (PLEG): container finished" podID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerID="508946ceb08ebfaac53d6368df470c66ba022845dacab73155f5b47e10a6f601" exitCode=60
Nov 01 04:00:01.959632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:01.959580    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" event={"ID":"2258d228-bef4-47e4-b4ca-d4ca466fedcf","Type":"ContainerDied","Data":"508946ceb08ebfaac53d6368df470c66ba022845dacab73155f5b47e10a6f601"}
Nov 01 04:00:01.959756 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:01.959740    4882 scope.go:122] "RemoveContainer" containerID="508946ceb08ebfaac53d6368df470c66ba022845dacab73155f5b47e10a6f601"
Nov 01 04:00:01.961352 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:01.961328    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" event={"ID":"08587f3b-3fdb-4888-b96a-46472d3e537f","Type":"ContainerStarted","Data":"b7c96ad0ddde618def6110ab60612cc9a121568d3b4244305ee64420dc4fd3dd"}
Nov 01 04:00:01.961413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:01.961372    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" event={"ID":"08587f3b-3fdb-4888-b96a-46472d3e537f","Type":"ContainerStarted","Data":"2bfb8adfb45f5d4a0fe3942508c544207c8c1c23088ee784091c1041e84da1ea"}
Nov 01 04:00:02.170437 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.170375    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" podStartSLOduration=2.170362429 podStartE2EDuration="2.170362429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:02.168627679 +0000 UTC m=+1070.730660696" watchObservedRunningTime="2026-11-01 04:00:02.170362429 +0000 UTC m=+1070.732395483"
Nov 01 04:00:02.970760 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.970727    4882 generic.go:356] "Generic (PLEG): container finished" podID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerID="c32a421e1d11613a2ebe42e404a4b82a32c9646b1db81886e18f9b99785c3030" exitCode=60
Nov 01 04:00:02.971161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.970807    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6" event={"ID":"dc6520c6-1e47-43e5-b2fa-11a77f00d501","Type":"ContainerDied","Data":"c32a421e1d11613a2ebe42e404a4b82a32c9646b1db81886e18f9b99785c3030"}
Nov 01 04:00:02.971161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.970867    4882 scope.go:122] "RemoveContainer" containerID="b64521247b1dc794822c95e908f149cf424fefc58544347f2428b08564e6960e"
Nov 01 04:00:02.971161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.971137    4882 scope.go:122] "RemoveContainer" containerID="c32a421e1d11613a2ebe42e404a4b82a32c9646b1db81886e18f9b99785c3030"
Nov 01 04:00:02.971491 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:02.971461    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6_odh-ai-gateway-infra(dc6520c6-1e47-43e5-b2fa-11a77f00d501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501"
Nov 01 04:00:02.972945 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.972920    4882 generic.go:356] "Generic (PLEG): container finished" podID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerID="1880b3869bdf9ef57ee08f680e21c599700f704de5bfc081ff34a0fc520d0876" exitCode=60
Nov 01 04:00:02.973053 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.972995    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x" event={"ID":"b3043c5b-899e-4e4d-b879-3b3eb798cf04","Type":"ContainerDied","Data":"1880b3869bdf9ef57ee08f680e21c599700f704de5bfc081ff34a0fc520d0876"}
Nov 01 04:00:02.973291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.973273    4882 scope.go:122] "RemoveContainer" containerID="1880b3869bdf9ef57ee08f680e21c599700f704de5bfc081ff34a0fc520d0876"
Nov 01 04:00:02.973553 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:02.973531    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891760-n9j9x_odh-ai-gateway-infra(b3043c5b-899e-4e4d-b879-3b3eb798cf04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04"
Nov 01 04:00:02.974881 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.974852    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" event={"ID":"2258d228-bef4-47e4-b4ca-d4ca466fedcf","Type":"ContainerStarted","Data":"d5c786df3fd4f28756522bcf588c1dd2a535daa37b2f36561828842a5ddf7bab"}
Nov 01 04:00:02.975114 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.975098    4882 scope.go:122] "RemoveContainer" containerID="d5c786df3fd4f28756522bcf588c1dd2a535daa37b2f36561828842a5ddf7bab"
Nov 01 04:00:02.975383 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:02.975361    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68_odh-ai-gateway-infra(2258d228-bef4-47e4-b4ca-d4ca466fedcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf"
Nov 01 04:00:02.976340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.976318    4882 generic.go:356] "Generic (PLEG): container finished" podID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerID="b7c96ad0ddde618def6110ab60612cc9a121568d3b4244305ee64420dc4fd3dd" exitCode=60
Nov 01 04:00:02.976421 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.976403    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" event={"ID":"08587f3b-3fdb-4888-b96a-46472d3e537f","Type":"ContainerDied","Data":"b7c96ad0ddde618def6110ab60612cc9a121568d3b4244305ee64420dc4fd3dd"}
Nov 01 04:00:02.976583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:02.976569    4882 scope.go:122] "RemoveContainer" containerID="b7c96ad0ddde618def6110ab60612cc9a121568d3b4244305ee64420dc4fd3dd"
Nov 01 04:00:03.058901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:03.058881    4882 scope.go:122] "RemoveContainer" containerID="228c1b7869fc7ffcdd269468a39e4633fd9dd3f2ffb8e22d491e71813a5f1b52"
Nov 01 04:00:03.990317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:03.990282    4882 scope.go:122] "RemoveContainer" containerID="c32a421e1d11613a2ebe42e404a4b82a32c9646b1db81886e18f9b99785c3030"
Nov 01 04:00:03.991009 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:03.990684    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6_odh-ai-gateway-infra(dc6520c6-1e47-43e5-b2fa-11a77f00d501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501"
Nov 01 04:00:03.993393 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:03.993364    4882 scope.go:122] "RemoveContainer" containerID="1880b3869bdf9ef57ee08f680e21c599700f704de5bfc081ff34a0fc520d0876"
Nov 01 04:00:03.993765 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:03.993731    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891760-n9j9x_odh-ai-gateway-infra(b3043c5b-899e-4e4d-b879-3b3eb798cf04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04"
Nov 01 04:00:03.995488 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:03.995461    4882 generic.go:356] "Generic (PLEG): container finished" podID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerID="d5c786df3fd4f28756522bcf588c1dd2a535daa37b2f36561828842a5ddf7bab" exitCode=60
Nov 01 04:00:03.995605 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:03.995556    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" event={"ID":"2258d228-bef4-47e4-b4ca-d4ca466fedcf","Type":"ContainerDied","Data":"d5c786df3fd4f28756522bcf588c1dd2a535daa37b2f36561828842a5ddf7bab"}
Nov 01 04:00:03.995676 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:03.995666    4882 scope.go:122] "RemoveContainer" containerID="508946ceb08ebfaac53d6368df470c66ba022845dacab73155f5b47e10a6f601"
Nov 01 04:00:03.996176 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:03.996030    4882 scope.go:122] "RemoveContainer" containerID="d5c786df3fd4f28756522bcf588c1dd2a535daa37b2f36561828842a5ddf7bab"
Nov 01 04:00:03.996358 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:03.996333    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68_odh-ai-gateway-infra(2258d228-bef4-47e4-b4ca-d4ca466fedcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf"
Nov 01 04:00:03.998173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:03.998147    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" event={"ID":"08587f3b-3fdb-4888-b96a-46472d3e537f","Type":"ContainerStarted","Data":"150191fd89f6ce122275b2f9759cfb3e9a24d8294277c5585c252ea31d135653"}
Nov 01 04:00:05.031659 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:05.031628    4882 scope.go:122] "RemoveContainer" containerID="d5c786df3fd4f28756522bcf588c1dd2a535daa37b2f36561828842a5ddf7bab"
Nov 01 04:00:05.033036 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:05.033000    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68_odh-ai-gateway-infra(2258d228-bef4-47e4-b4ca-d4ca466fedcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf"
Nov 01 04:00:05.034844 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:05.034804    4882 generic.go:356] "Generic (PLEG): container finished" podID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerID="150191fd89f6ce122275b2f9759cfb3e9a24d8294277c5585c252ea31d135653" exitCode=60
Nov 01 04:00:05.035075 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:05.035048    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" event={"ID":"08587f3b-3fdb-4888-b96a-46472d3e537f","Type":"ContainerDied","Data":"150191fd89f6ce122275b2f9759cfb3e9a24d8294277c5585c252ea31d135653"}
Nov 01 04:00:05.035225 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:05.035210    4882 scope.go:122] "RemoveContainer" containerID="b7c96ad0ddde618def6110ab60612cc9a121568d3b4244305ee64420dc4fd3dd"
Nov 01 04:00:05.035885 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:05.035855    4882 scope.go:122] "RemoveContainer" containerID="150191fd89f6ce122275b2f9759cfb3e9a24d8294277c5585c252ea31d135653"
Nov 01 04:00:05.036391 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:05.036360    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5_odh-ai-gateway-infra(08587f3b-3fdb-4888-b96a-46472d3e537f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f"
Nov 01 04:00:05.473453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:05.471544    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flvv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:00:05.473453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:05.471615    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:00:05.508453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:05.504547    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jxfk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:00:05.508453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:05.504615    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:00:06.045690 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:06.045659    4882 scope.go:122] "RemoveContainer" containerID="150191fd89f6ce122275b2f9759cfb3e9a24d8294277c5585c252ea31d135653"
Nov 01 04:00:06.046225 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:06.045936    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5_odh-ai-gateway-infra(08587f3b-3fdb-4888-b96a-46472d3e537f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f"
Nov 01 04:00:11.789373 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:11.789339    4882 scope.go:122] "RemoveContainer" containerID="4c207a2baa9c432695eec079f58bb27de3c935722de994a3bce772d38a10d2ef"
Nov 01 04:00:11.799398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:11.799377    4882 scope.go:122] "RemoveContainer" containerID="c0675fc51cc965356375d9509d3299df94bce126885ce8dfd49b2b3295e9939f"
Nov 01 04:00:11.807912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:11.807893    4882 scope.go:122] "RemoveContainer" containerID="325f228409c9c4976bc991d3603c89f69efb3ead349a494e3b88c07ebad32035"
Nov 01 04:00:11.817110 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:11.817091    4882 scope.go:122] "RemoveContainer" containerID="2ca2546ef529200bbf761afd1a3f8b664ddf1f1d8e09928790a91b40822c2d01"
Nov 01 04:00:11.825948 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:11.825923    4882 scope.go:122] "RemoveContainer" containerID="0c1166e0cab500dfba3c3adfd86439e6f36de9f983ddfd01870c1d88f18e7ec6"
Nov 01 04:00:15.592560 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:15.592526    4882 scope.go:122] "RemoveContainer" containerID="c32a421e1d11613a2ebe42e404a4b82a32c9646b1db81886e18f9b99785c3030"
Nov 01 04:00:16.116459 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:16.116403    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6" event={"ID":"dc6520c6-1e47-43e5-b2fa-11a77f00d501","Type":"ContainerStarted","Data":"25566c448469fd31e6fb14036a9a64c7d57deeaae3c9114504218ce8b123efe2"}
Nov 01 04:00:16.227468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:16.227403    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6" podStartSLOduration=16.227388074 podStartE2EDuration="16.227388074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:16.224283406 +0000 UTC m=+1084.786316433" watchObservedRunningTime="2026-11-01 04:00:16.227388074 +0000 UTC m=+1084.789421091"
Nov 01 04:00:16.593039 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:16.592953    4882 scope.go:122] "RemoveContainer" containerID="d5c786df3fd4f28756522bcf588c1dd2a535daa37b2f36561828842a5ddf7bab"
Nov 01 04:00:17.130877 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:17.130784    4882 generic.go:356] "Generic (PLEG): container finished" podID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerID="25566c448469fd31e6fb14036a9a64c7d57deeaae3c9114504218ce8b123efe2" exitCode=60
Nov 01 04:00:17.131099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:17.131080    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6" event={"ID":"dc6520c6-1e47-43e5-b2fa-11a77f00d501","Type":"ContainerDied","Data":"25566c448469fd31e6fb14036a9a64c7d57deeaae3c9114504218ce8b123efe2"}
Nov 01 04:00:17.131189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:17.131170    4882 scope.go:122] "RemoveContainer" containerID="c32a421e1d11613a2ebe42e404a4b82a32c9646b1db81886e18f9b99785c3030"
Nov 01 04:00:17.131541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:17.131521    4882 scope.go:122] "RemoveContainer" containerID="25566c448469fd31e6fb14036a9a64c7d57deeaae3c9114504218ce8b123efe2"
Nov 01 04:00:17.131957 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:17.131935    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6_odh-ai-gateway-infra(dc6520c6-1e47-43e5-b2fa-11a77f00d501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501"
Nov 01 04:00:17.133232 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:17.133211    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" event={"ID":"2258d228-bef4-47e4-b4ca-d4ca466fedcf","Type":"ContainerStarted","Data":"88a14af4a86fd47bcd539c30d4603dc53acbc6454a6d1c367dd5c11bd6e156f8"}
Nov 01 04:00:17.235580 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:17.235548    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"]
Nov 01 04:00:17.373739 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:17.373686    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" podStartSLOduration=17.373671341 podStartE2EDuration="17.373671341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:17.371019641 +0000 UTC m=+1085.933052658" watchObservedRunningTime="2026-11-01 04:00:17.373671341 +0000 UTC m=+1085.935704358"
Nov 01 04:00:17.593235 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:17.593200    4882 scope.go:122] "RemoveContainer" containerID="1880b3869bdf9ef57ee08f680e21c599700f704de5bfc081ff34a0fc520d0876"
Nov 01 04:00:18.143021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.142974    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x" event={"ID":"b3043c5b-899e-4e4d-b879-3b3eb798cf04","Type":"ContainerStarted","Data":"5700daa7fdb77f30d538338d5ccb889d540e263620e34866caaf2d9a08e04f9a"}
Nov 01 04:00:18.145019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.144993    4882 generic.go:356] "Generic (PLEG): container finished" podID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerID="88a14af4a86fd47bcd539c30d4603dc53acbc6454a6d1c367dd5c11bd6e156f8" exitCode=60
Nov 01 04:00:18.145134 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.145061    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" event={"ID":"2258d228-bef4-47e4-b4ca-d4ca466fedcf","Type":"ContainerDied","Data":"88a14af4a86fd47bcd539c30d4603dc53acbc6454a6d1c367dd5c11bd6e156f8"}
Nov 01 04:00:18.145134 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.145127    4882 scope.go:122] "RemoveContainer" containerID="d5c786df3fd4f28756522bcf588c1dd2a535daa37b2f36561828842a5ddf7bab"
Nov 01 04:00:18.145459 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.145443    4882 scope.go:122] "RemoveContainer" containerID="88a14af4a86fd47bcd539c30d4603dc53acbc6454a6d1c367dd5c11bd6e156f8"
Nov 01 04:00:18.145771 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:18.145748    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68_odh-ai-gateway-infra(2258d228-bef4-47e4-b4ca-d4ca466fedcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf"
Nov 01 04:00:18.297204 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.297177    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"
Nov 01 04:00:18.359808 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.359759    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x" podStartSLOduration=18.359745269 podStartE2EDuration="18.359745269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:18.355113939 +0000 UTC m=+1086.917146956" watchObservedRunningTime="2026-11-01 04:00:18.359745269 +0000 UTC m=+1086.921778295"
Nov 01 04:00:18.381782 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.381754    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"]
Nov 01 04:00:18.392469 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.392365    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6520c6-1e47-43e5-b2fa-11a77f00d501-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6520c6-1e47-43e5-b2fa-11a77f00d501-maas-api-service-ca\") pod \"dc6520c6-1e47-43e5-b2fa-11a77f00d501\" (UID: \"dc6520c6-1e47-43e5-b2fa-11a77f00d501\") "
Nov 01 04:00:18.392469 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.392448    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6520c6-1e47-43e5-b2fa-11a77f00d501-kube-api-access-hssmg\" (UniqueName: \"kubernetes.io/projected/dc6520c6-1e47-43e5-b2fa-11a77f00d501-kube-api-access-hssmg\") pod \"dc6520c6-1e47-43e5-b2fa-11a77f00d501\" (UID: \"dc6520c6-1e47-43e5-b2fa-11a77f00d501\") "
Nov 01 04:00:18.392719 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.392694    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6520c6-1e47-43e5-b2fa-11a77f00d501-maas-api-service-ca" pod "dc6520c6-1e47-43e5-b2fa-11a77f00d501" (UID: "dc6520c6-1e47-43e5-b2fa-11a77f00d501"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:00:18.393014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.392996    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6520c6-1e47-43e5-b2fa-11a77f00d501-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:00:18.394408 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.394387    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6520c6-1e47-43e5-b2fa-11a77f00d501-kube-api-access-hssmg" pod "dc6520c6-1e47-43e5-b2fa-11a77f00d501" (UID: "dc6520c6-1e47-43e5-b2fa-11a77f00d501"). InnerVolumeSpecName "kube-api-access-hssmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:00:18.494562 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.494527    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hssmg\" (UniqueName: \"kubernetes.io/projected/dc6520c6-1e47-43e5-b2fa-11a77f00d501-kube-api-access-hssmg\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:00:18.596115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.596080    4882 scope.go:122] "RemoveContainer" containerID="150191fd89f6ce122275b2f9759cfb3e9a24d8294277c5585c252ea31d135653"
Nov 01 04:00:18.599886 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.599858    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"]
Nov 01 04:00:18.600586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.600567    4882 cpu_manager.go:414] "containerMap: removing container" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerName="cleanup"
Nov 01 04:00:18.600645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.600589    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerName="cleanup"
Nov 01 04:00:18.600645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.600641    4882 cpu_manager.go:414] "containerMap: removing container" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerName="cleanup"
Nov 01 04:00:18.600772 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.600649    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerName="cleanup"
Nov 01 04:00:18.600825 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.600800    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerName="cleanup"
Nov 01 04:00:18.600825 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.600818    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerName="cleanup"
Nov 01 04:00:18.605914 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.605887    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.698472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.698401    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.698667 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.698501    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.698667 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.698559    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8hvk\" (UniqueName: \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-kube-api-access-r8hvk\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.698781 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.698716    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.698916 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.698870    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.699212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.699150    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.699335 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.699315    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.699403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.699389    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.699522 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.699507    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fed9c9-bba5-4025-a338-8d7c7a831941-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.699712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.699686    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"]
Nov 01 04:00:18.800717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.800681    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.800878 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.800734    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.800878 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.800763    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.800878 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.800794    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.800878 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.800827    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.801113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.800893    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fed9c9-bba5-4025-a338-8d7c7a831941-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.801113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.800980    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.801113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.801016    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.801113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.801050    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8hvk\" (UniqueName: \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-kube-api-access-r8hvk\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.801298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.801126    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.801298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.801225    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.801453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.801417    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.801549 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.801526    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.801717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.801693    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fed9c9-bba5-4025-a338-8d7c7a831941-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.803298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.803266    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.803298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.803287    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.858280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.858252    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.858470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.858419    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8hvk\" (UniqueName: \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-kube-api-access-r8hvk\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:18.954103 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:18.954070    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:19.152751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.152722    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"
Nov 01 04:00:19.152939 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.152751    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6" event={"ID":"dc6520c6-1e47-43e5-b2fa-11a77f00d501","Type":"ContainerDied","Data":"83e5fa6045bbb556a50f35eb601915a2ef02e38cfbaa0941955517696b7831dc"}
Nov 01 04:00:19.152939 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.152812    4882 scope.go:122] "RemoveContainer" containerID="25566c448469fd31e6fb14036a9a64c7d57deeaae3c9114504218ce8b123efe2"
Nov 01 04:00:19.154757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.154731    4882 generic.go:356] "Generic (PLEG): container finished" podID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerID="5700daa7fdb77f30d538338d5ccb889d540e263620e34866caaf2d9a08e04f9a" exitCode=60
Nov 01 04:00:19.154843 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.154800    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x" event={"ID":"b3043c5b-899e-4e4d-b879-3b3eb798cf04","Type":"ContainerDied","Data":"5700daa7fdb77f30d538338d5ccb889d540e263620e34866caaf2d9a08e04f9a"}
Nov 01 04:00:19.155184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.155168    4882 scope.go:122] "RemoveContainer" containerID="5700daa7fdb77f30d538338d5ccb889d540e263620e34866caaf2d9a08e04f9a"
Nov 01 04:00:19.155494 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:19.155471    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891760-n9j9x_odh-ai-gateway-infra(b3043c5b-899e-4e4d-b879-3b3eb798cf04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04"
Nov 01 04:00:19.158743 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.158720    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" event={"ID":"08587f3b-3fdb-4888-b96a-46472d3e537f","Type":"ContainerStarted","Data":"78ab36c7297655b27122aa80a3d05def14638da960b6d37bf15620101f663b86"}
Nov 01 04:00:19.164486 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.164410    4882 scope.go:122] "RemoveContainer" containerID="1880b3869bdf9ef57ee08f680e21c599700f704de5bfc081ff34a0fc520d0876"
Nov 01 04:00:19.299505 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.299476    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"
Nov 01 04:00:19.368750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.368710    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"]
Nov 01 04:00:19.408039 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.408003    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2258d228-bef4-47e4-b4ca-d4ca466fedcf-kube-api-access-bszsb\" (UniqueName: \"kubernetes.io/projected/2258d228-bef4-47e4-b4ca-d4ca466fedcf-kube-api-access-bszsb\") pod \"2258d228-bef4-47e4-b4ca-d4ca466fedcf\" (UID: \"2258d228-bef4-47e4-b4ca-d4ca466fedcf\") "
Nov 01 04:00:19.408209 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.408106    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2258d228-bef4-47e4-b4ca-d4ca466fedcf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2258d228-bef4-47e4-b4ca-d4ca466fedcf-maas-api-service-ca\") pod \"2258d228-bef4-47e4-b4ca-d4ca466fedcf\" (UID: \"2258d228-bef4-47e4-b4ca-d4ca466fedcf\") "
Nov 01 04:00:19.408575 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.408539    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2258d228-bef4-47e4-b4ca-d4ca466fedcf-maas-api-service-ca" pod "2258d228-bef4-47e4-b4ca-d4ca466fedcf" (UID: "2258d228-bef4-47e4-b4ca-d4ca466fedcf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:00:19.425504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.425478    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"]
Nov 01 04:00:19.426621 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:00:19.426593    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fed9c9_bba5_4025_a338_8d7c7a831941.slice/crio-d441b36ebc34cbd5c322257226c108e01d82a7e15578991c42fde7a1e69b1d58 WatchSource:0}: Error finding container d441b36ebc34cbd5c322257226c108e01d82a7e15578991c42fde7a1e69b1d58: Status 404 returned error can't find the container with id d441b36ebc34cbd5c322257226c108e01d82a7e15578991c42fde7a1e69b1d58
Nov 01 04:00:19.429162 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.429134    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:00:19.429248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.429194    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:00:19.429248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.429223    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:00:19.462208 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.461898    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2258d228-bef4-47e4-b4ca-d4ca466fedcf-kube-api-access-bszsb" pod "2258d228-bef4-47e4-b4ca-d4ca466fedcf" (UID: "2258d228-bef4-47e4-b4ca-d4ca466fedcf"). InnerVolumeSpecName "kube-api-access-bszsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:00:19.509541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.509466    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bszsb\" (UniqueName: \"kubernetes.io/projected/2258d228-bef4-47e4-b4ca-d4ca466fedcf-kube-api-access-bszsb\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:00:19.509541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.509495    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2258d228-bef4-47e4-b4ca-d4ca466fedcf-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:00:19.522576 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.522547    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"]
Nov 01 04:00:19.527036 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.527012    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891760-9krj6"]
Nov 01 04:00:19.602667 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:19.602632    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" path="/var/lib/kubelet/pods/dc6520c6-1e47-43e5-b2fa-11a77f00d501/volumes"
Nov 01 04:00:20.173472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.173416    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68" event={"ID":"2258d228-bef4-47e4-b4ca-d4ca466fedcf","Type":"ContainerDied","Data":"33b8de09bd9965d5d6c06909bed6a4813ec7ebd26b17674bfca9d48c4e272ca4"}
Nov 01 04:00:20.173766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.173749    4882 scope.go:122] "RemoveContainer" containerID="88a14af4a86fd47bcd539c30d4603dc53acbc6454a6d1c367dd5c11bd6e156f8"
Nov 01 04:00:20.174047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.174032    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"
Nov 01 04:00:20.182574 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.182543    4882 generic.go:356] "Generic (PLEG): container finished" podID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerID="78ab36c7297655b27122aa80a3d05def14638da960b6d37bf15620101f663b86" exitCode=60
Nov 01 04:00:20.182842 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.182706    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" event={"ID":"08587f3b-3fdb-4888-b96a-46472d3e537f","Type":"ContainerDied","Data":"78ab36c7297655b27122aa80a3d05def14638da960b6d37bf15620101f663b86"}
Nov 01 04:00:20.184747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.184729    4882 scope.go:122] "RemoveContainer" containerID="78ab36c7297655b27122aa80a3d05def14638da960b6d37bf15620101f663b86"
Nov 01 04:00:20.189479 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.187401    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" event={"ID":"d8fed9c9-bba5-4025-a338-8d7c7a831941","Type":"ContainerStarted","Data":"4b0169f70fd676e471b6a85310c52f052c99e6cac6d286724238f630ee2a45cd"}
Nov 01 04:00:20.189686 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.189661    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" event={"ID":"d8fed9c9-bba5-4025-a338-8d7c7a831941","Type":"ContainerStarted","Data":"d441b36ebc34cbd5c322257226c108e01d82a7e15578991c42fde7a1e69b1d58"}
Nov 01 04:00:20.189907 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:20.189885    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5_odh-ai-gateway-infra(08587f3b-3fdb-4888-b96a-46472d3e537f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f"
Nov 01 04:00:20.206659 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.203948    4882 scope.go:122] "RemoveContainer" containerID="150191fd89f6ce122275b2f9759cfb3e9a24d8294277c5585c252ea31d135653"
Nov 01 04:00:20.394837 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.394805    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"
Nov 01 04:00:20.409593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.409539    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" podStartSLOduration=2.409524927 podStartE2EDuration="2.409524927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:20.408229978 +0000 UTC m=+1088.970262993" watchObservedRunningTime="2026-11-01 04:00:20.409524927 +0000 UTC m=+1088.971557944"
Nov 01 04:00:20.422094 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.422024    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3043c5b-899e-4e4d-b879-3b3eb798cf04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3043c5b-899e-4e4d-b879-3b3eb798cf04-maas-api-service-ca\") pod \"b3043c5b-899e-4e4d-b879-3b3eb798cf04\" (UID: \"b3043c5b-899e-4e4d-b879-3b3eb798cf04\") "
Nov 01 04:00:20.422209 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.422172    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3043c5b-899e-4e4d-b879-3b3eb798cf04-kube-api-access-8rhqv\" (UniqueName: \"kubernetes.io/projected/b3043c5b-899e-4e4d-b879-3b3eb798cf04-kube-api-access-8rhqv\") pod \"b3043c5b-899e-4e4d-b879-3b3eb798cf04\" (UID: \"b3043c5b-899e-4e4d-b879-3b3eb798cf04\") "
Nov 01 04:00:20.422333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.422309    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3043c5b-899e-4e4d-b879-3b3eb798cf04-maas-api-service-ca" pod "b3043c5b-899e-4e4d-b879-3b3eb798cf04" (UID: "b3043c5b-899e-4e4d-b879-3b3eb798cf04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:00:20.422716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.422698    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3043c5b-899e-4e4d-b879-3b3eb798cf04-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:00:20.424245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.424226    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3043c5b-899e-4e4d-b879-3b3eb798cf04-kube-api-access-8rhqv" pod "b3043c5b-899e-4e4d-b879-3b3eb798cf04" (UID: "b3043c5b-899e-4e4d-b879-3b3eb798cf04"). InnerVolumeSpecName "kube-api-access-8rhqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:00:20.523683 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.523646    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rhqv\" (UniqueName: \"kubernetes.io/projected/b3043c5b-899e-4e4d-b879-3b3eb798cf04-kube-api-access-8rhqv\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:00:20.632951 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.632911    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"]
Nov 01 04:00:20.639125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.639087    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891760-w2h68"]
Nov 01 04:00:20.689515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.681480    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"]
Nov 01 04:00:20.978508 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.967775    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:20.978508 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.971549    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 01 04:00:20.978508 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:20.971607    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 01 04:00:21.221966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.221938    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"
Nov 01 04:00:21.235507 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.234639    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x" event={"ID":"b3043c5b-899e-4e4d-b879-3b3eb798cf04","Type":"ContainerDied","Data":"70b2b9ed40691c92941b46ce7b4273ac560742da5c73d61a632dc1989617b6fd"}
Nov 01 04:00:21.235507 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.234717    4882 scope.go:122] "RemoveContainer" containerID="5700daa7fdb77f30d538338d5ccb889d540e263620e34866caaf2d9a08e04f9a"
Nov 01 04:00:21.528659 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.528545    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"]
Nov 01 04:00:21.534514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.534486    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-n9j9x"]
Nov 01 04:00:21.705356 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.705323    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" path="/var/lib/kubelet/pods/2258d228-bef4-47e4-b4ca-d4ca466fedcf/volumes"
Nov 01 04:00:21.706750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.706715    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" path="/var/lib/kubelet/pods/b3043c5b-899e-4e4d-b879-3b3eb798cf04/volumes"
Nov 01 04:00:21.773450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.761231    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"
Nov 01 04:00:21.857273 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.857174    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08587f3b-3fdb-4888-b96a-46472d3e537f-kube-api-access-xlnwq\" (UniqueName: \"kubernetes.io/projected/08587f3b-3fdb-4888-b96a-46472d3e537f-kube-api-access-xlnwq\") pod \"08587f3b-3fdb-4888-b96a-46472d3e537f\" (UID: \"08587f3b-3fdb-4888-b96a-46472d3e537f\") "
Nov 01 04:00:21.857997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.857957    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08587f3b-3fdb-4888-b96a-46472d3e537f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08587f3b-3fdb-4888-b96a-46472d3e537f-maas-api-service-ca\") pod \"08587f3b-3fdb-4888-b96a-46472d3e537f\" (UID: \"08587f3b-3fdb-4888-b96a-46472d3e537f\") "
Nov 01 04:00:21.858530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.858501    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08587f3b-3fdb-4888-b96a-46472d3e537f-maas-api-service-ca" pod "08587f3b-3fdb-4888-b96a-46472d3e537f" (UID: "08587f3b-3fdb-4888-b96a-46472d3e537f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:00:21.860351 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.860326    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08587f3b-3fdb-4888-b96a-46472d3e537f-kube-api-access-xlnwq" pod "08587f3b-3fdb-4888-b96a-46472d3e537f" (UID: "08587f3b-3fdb-4888-b96a-46472d3e537f"). InnerVolumeSpecName "kube-api-access-xlnwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:00:21.965458 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.960808    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 01 04:00:21.965458 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.960872    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 01 04:00:21.968269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.968233    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlnwq\" (UniqueName: \"kubernetes.io/projected/08587f3b-3fdb-4888-b96a-46472d3e537f-kube-api-access-xlnwq\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:00:21.990796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:21.990768    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08587f3b-3fdb-4888-b96a-46472d3e537f-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:00:22.648927 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:22.648891    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"
Nov 01 04:00:22.703329 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:22.667477    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5" event={"ID":"08587f3b-3fdb-4888-b96a-46472d3e537f","Type":"ContainerDied","Data":"2bfb8adfb45f5d4a0fe3942508c544207c8c1c23088ee784091c1041e84da1ea"}
Nov 01 04:00:22.706557 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:22.706529    4882 scope.go:122] "RemoveContainer" containerID="78ab36c7297655b27122aa80a3d05def14638da960b6d37bf15620101f663b86"
Nov 01 04:00:22.875815 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:22.875679    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"]
Nov 01 04:00:22.881130 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:22.881100    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891760-9v5q5"]
Nov 01 04:00:22.956222 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:22.956187    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 01 04:00:22.956496 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:22.956466    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 01 04:00:23.676450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:23.676149    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f" path="/var/lib/kubelet/pods/08587f3b-3fdb-4888-b96a-46472d3e537f/volumes"
Nov 01 04:00:23.959450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:23.959089    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 01 04:00:23.959450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:23.959165    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 01 04:00:24.954835 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:24.954806    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 01 04:00:24.954979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:24.954857    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 01 04:00:25.954589 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:25.954558    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 01 04:00:25.955054 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:25.954613    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 01 04:00:27.091097 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:27.091065    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:27.091980 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:27.091950    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:27.093242 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:27.093221    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:00:28.723736 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:28.723686    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:00:36.599647 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.599608    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"]
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600090    4882 cpu_manager.go:414] "containerMap: removing container" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600100    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600112    4882 cpu_manager.go:414] "containerMap: removing container" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600117    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600125    4882 cpu_manager.go:414] "containerMap: removing container" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600131    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600145    4882 cpu_manager.go:414] "containerMap: removing container" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600151    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600158    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600162    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600172    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600176    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600182    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600187    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600200    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600204    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600304    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600312    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600318    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600324    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600330    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600335    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600342    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6520c6-1e47-43e5-b2fa-11a77f00d501" containerName="cleanup"
Nov 01 04:00:36.602439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.600352    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerName="cleanup"
Nov 01 04:00:36.603661 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.603642    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.658818 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.658776    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"]
Nov 01 04:00:36.701096 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.701058    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-162191\"/\"openshift-service-ca.crt\""
Nov 01 04:00:36.701096 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.701058    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-162191\"/\"kube-root-ca.crt\""
Nov 01 04:00:36.701337 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.701145    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-162191\"/\"distinct-w0-162191-kserve-self-signed-certs\""
Nov 01 04:00:36.701542 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.701530    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-162191\"/\"default-dockercfg-2zg92\""
Nov 01 04:00:36.723714 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.723688    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-home\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.723841 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.723717    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnhmf\" (UniqueName: \"kubernetes.io/projected/210fc38c-871d-4b42-8e35-e3407510f122-kube-api-access-rnhmf\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.723841 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.723738    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210fc38c-871d-4b42-8e35-e3407510f122-tls-certs\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.724165 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.724138    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-tmp-dir\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.724252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.724240    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-model-cache\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.724286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.724264    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-dshm\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.825475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.825422    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-tmp-dir\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.825651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.825507    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-model-cache\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.825651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.825537    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-dshm\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.825651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.825573    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-home\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.825651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.825592    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnhmf\" (UniqueName: \"kubernetes.io/projected/210fc38c-871d-4b42-8e35-e3407510f122-kube-api-access-rnhmf\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.825651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.825616    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210fc38c-871d-4b42-8e35-e3407510f122-tls-certs\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.825962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.825934    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-tmp-dir\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.826027 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.825974    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-model-cache\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.826083 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.826028    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-home\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.827792 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.827769    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-dshm\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.828163 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.828142    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210fc38c-871d-4b42-8e35-e3407510f122-tls-certs\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.883041 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.882967    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnhmf\" (UniqueName: \"kubernetes.io/projected/210fc38c-871d-4b42-8e35-e3407510f122-kube-api-access-rnhmf\") pod \"distinct-w0-162191-kserve-6695bf8cdf-8w8tw\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") " pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:36.915703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:36.915665    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:37.302257 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:37.302228    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"]
Nov 01 04:00:37.303979 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:00:37.303951    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod210fc38c_871d_4b42_8e35_e3407510f122.slice/crio-2f0956e93e52d2a5c4549af78d1104153c9726f1b2411099d7011ce0a99af1e9 WatchSource:0}: Error finding container 2f0956e93e52d2a5c4549af78d1104153c9726f1b2411099d7011ce0a99af1e9: Status 404 returned error can't find the container with id 2f0956e93e52d2a5c4549af78d1104153c9726f1b2411099d7011ce0a99af1e9
Nov 01 04:00:37.784313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:37.784276    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" event={"ID":"210fc38c-871d-4b42-8e35-e3407510f122","Type":"ContainerStarted","Data":"e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e"}
Nov 01 04:00:37.784313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:37.784323    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" event={"ID":"210fc38c-871d-4b42-8e35-e3407510f122","Type":"ContainerStarted","Data":"2f0956e93e52d2a5c4549af78d1104153c9726f1b2411099d7011ce0a99af1e9"}
Nov 01 04:00:37.895104 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:37.895051    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podStartSLOduration=1.8950379979999998 podStartE2EDuration="1.895037998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:37.892864623 +0000 UTC m=+1106.454897639" watchObservedRunningTime="2026-11-01 04:00:37.895037998 +0000 UTC m=+1106.457071015"
Nov 01 04:00:37.916821 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:37.916791    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:44.112517 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.112479    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"]
Nov 01 04:00:44.114238 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.114209    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerName="cleanup"
Nov 01 04:00:44.114410 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.114397    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3043c5b-899e-4e4d-b879-3b3eb798cf04" containerName="cleanup"
Nov 01 04:00:44.114582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.114569    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerName="cleanup"
Nov 01 04:00:44.114665 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.114654    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerName="cleanup"
Nov 01 04:00:44.115623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.115600    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="08587f3b-3fdb-4888-b96a-46472d3e537f" containerName="cleanup"
Nov 01 04:00:44.115783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.115765    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2258d228-bef4-47e4-b4ca-d4ca466fedcf" containerName="cleanup"
Nov 01 04:00:44.120964 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.120938    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.205137 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.205098    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"]
Nov 01 04:00:44.208827 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.208792    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.208989 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.208842    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.208989 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.208889    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948bb297-3490-4419-9cbd-5e11c367c538-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.208989 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.208915    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.208989 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.208953    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948bb297-3490-4419-9cbd-5e11c367c538-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.209418 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.208996    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.209418 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.209057    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.209418 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.209081    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.209418 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.209114    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn58w\" (UniqueName: \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-kube-api-access-nn58w\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310038 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.309998    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310038 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310040    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310167    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948bb297-3490-4419-9cbd-5e11c367c538-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310201    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310351 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310252    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948bb297-3490-4419-9cbd-5e11c367c538-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310351 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310305    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310370    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310389    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310394    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310464    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn58w\" (UniqueName: \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-kube-api-access-nn58w\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310714 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310476    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310714 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310667    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310858 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310835    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.310985 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.310969    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948bb297-3490-4419-9cbd-5e11c367c538-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.312639 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.312615    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.312933 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.312914    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948bb297-3490-4419-9cbd-5e11c367c538-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.368173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.368102    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.368308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.368174    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn58w\" (UniqueName: \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-kube-api-access-nn58w\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.434588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.434558    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:44.855245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.855221    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"]
Nov 01 04:00:44.856849 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:00:44.856806    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948bb297_3490_4419_9cbd_5e11c367c538.slice/crio-4d33351925a8e39bb9d1deeb24c8dd16ccfe62ff8c0aada0123b4c3dce3bbb42 WatchSource:0}: Error finding container 4d33351925a8e39bb9d1deeb24c8dd16ccfe62ff8c0aada0123b4c3dce3bbb42: Status 404 returned error can't find the container with id 4d33351925a8e39bb9d1deeb24c8dd16ccfe62ff8c0aada0123b4c3dce3bbb42
Nov 01 04:00:44.859388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.859355    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:00:44.859463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.859440    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:00:44.859511 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:44.859473    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:00:45.836311 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:45.836275    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" event={"ID":"948bb297-3490-4419-9cbd-5e11c367c538","Type":"ContainerStarted","Data":"90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d"}
Nov 01 04:00:45.836762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:45.836326    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" event={"ID":"948bb297-3490-4419-9cbd-5e11c367c538","Type":"ContainerStarted","Data":"4d33351925a8e39bb9d1deeb24c8dd16ccfe62ff8c0aada0123b4c3dce3bbb42"}
Nov 01 04:00:46.043173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:46.043113    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" podStartSLOduration=2.043098346 podStartE2EDuration="2.043098346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:45.996170027 +0000 UTC m=+1114.558203045" watchObservedRunningTime="2026-11-01 04:00:46.043098346 +0000 UTC m=+1114.605131362"
Nov 01 04:00:46.435340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:46.435304    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:46.436580 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:46.436558    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 01 04:00:46.436632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:46.436605    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 01 04:00:46.916191 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:46.916100    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:46.929067 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:46.929045    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:47.435088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:47.435051    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 01 04:00:47.435296 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:47.435113    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 01 04:00:47.860576 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:47.860495    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:00:48.435222 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:48.435190    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 01 04:00:48.435702 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:48.435269    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 01 04:00:49.435013 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:49.434981    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 01 04:00:49.435199 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:49.435032    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 01 04:00:50.435468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:50.435361    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 01 04:00:50.435468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:50.435451    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 01 04:00:51.435631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:51.435592    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 01 04:00:51.436224 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:51.436189    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 01 04:00:51.864500 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:51.863667    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"]
Nov 01 04:00:51.871977 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:51.871948    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:51.923179 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:51.923144    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"]
Nov 01 04:00:51.962993 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:51.962959    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:00:51.963830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:51.963766    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:00:52.022184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.022148    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81fc914-nqg85\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:52.022388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.022241    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9wz\" (UniqueName: \"kubernetes.io/projected/9d7875a8-9210-4eda-8206-20ee8820c99a-kube-api-access-xq9wz\") pod \"maas-api-revoke-sub-c81fc914-nqg85\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:52.022509 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.022411    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81fc914-nqg85\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:52.123656 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.123558    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81fc914-nqg85\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:52.123827 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.123683    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81fc914-nqg85\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:52.123827 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.123759    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9wz\" (UniqueName: \"kubernetes.io/projected/9d7875a8-9210-4eda-8206-20ee8820c99a-kube-api-access-xq9wz\") pod \"maas-api-revoke-sub-c81fc914-nqg85\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:52.124332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.124309    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81fc914-nqg85\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:52.126784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.126760    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81fc914-nqg85\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:52.182202 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.182166    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9wz\" (UniqueName: \"kubernetes.io/projected/9d7875a8-9210-4eda-8206-20ee8820c99a-kube-api-access-xq9wz\") pod \"maas-api-revoke-sub-c81fc914-nqg85\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:52.189803 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.189775    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:52.439570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.439538    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:52.439935 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.439834    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:52.440479 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.440462    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:00:52.604246 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.604155    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"]
Nov 01 04:00:52.886119 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.886092    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85" event={"ID":"9d7875a8-9210-4eda-8206-20ee8820c99a","Type":"ContainerStarted","Data":"d206380fe68f546ba67635fa5e7ba2dd6a80d9737f3374a7f80cdb24ab87120a"}
Nov 01 04:00:52.886227 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.886143    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85" event={"ID":"9d7875a8-9210-4eda-8206-20ee8820c99a","Type":"ContainerStarted","Data":"ace9e7d89cd91c5ad76ba914bc5114fe9597ed6821ab375446375f089cb19f3c"}
Nov 01 04:00:52.995883 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:52.995840    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85" podStartSLOduration=1.995828334 podStartE2EDuration="1.995828334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:52.993396077 +0000 UTC m=+1121.555429094" watchObservedRunningTime="2026-11-01 04:00:52.995828334 +0000 UTC m=+1121.557861351"
Nov 01 04:00:53.894043 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:53.894004    4882 generic.go:356] "Generic (PLEG): container finished" podID="9d7875a8-9210-4eda-8206-20ee8820c99a" containerID="d206380fe68f546ba67635fa5e7ba2dd6a80d9737f3374a7f80cdb24ab87120a" exitCode=0
Nov 01 04:00:53.894500 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:53.894050    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85" event={"ID":"9d7875a8-9210-4eda-8206-20ee8820c99a","Type":"ContainerDied","Data":"d206380fe68f546ba67635fa5e7ba2dd6a80d9737f3374a7f80cdb24ab87120a"}
Nov 01 04:00:55.057865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.057838    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:55.165016 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.164932    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-openshift-ca\") pod \"9d7875a8-9210-4eda-8206-20ee8820c99a\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") "
Nov 01 04:00:55.165168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.165128    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7875a8-9210-4eda-8206-20ee8820c99a-kube-api-access-xq9wz\" (UniqueName: \"kubernetes.io/projected/9d7875a8-9210-4eda-8206-20ee8820c99a-kube-api-access-xq9wz\") pod \"9d7875a8-9210-4eda-8206-20ee8820c99a\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") "
Nov 01 04:00:55.165214 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.165200    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-xks-ca\") pod \"9d7875a8-9210-4eda-8206-20ee8820c99a\" (UID: \"9d7875a8-9210-4eda-8206-20ee8820c99a\") "
Nov 01 04:00:55.165327 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.165297    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-openshift-ca" pod "9d7875a8-9210-4eda-8206-20ee8820c99a" (UID: "9d7875a8-9210-4eda-8206-20ee8820c99a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:00:55.165664 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.165643    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:00:55.167608 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.167577    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-xks-ca" pod "9d7875a8-9210-4eda-8206-20ee8820c99a" (UID: "9d7875a8-9210-4eda-8206-20ee8820c99a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:00:55.168588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.168562    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7875a8-9210-4eda-8206-20ee8820c99a-kube-api-access-xq9wz" pod "9d7875a8-9210-4eda-8206-20ee8820c99a" (UID: "9d7875a8-9210-4eda-8206-20ee8820c99a"). InnerVolumeSpecName "kube-api-access-xq9wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:00:55.267260 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.267229    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9wz\" (UniqueName: \"kubernetes.io/projected/9d7875a8-9210-4eda-8206-20ee8820c99a-kube-api-access-xq9wz\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:00:55.267260 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.267259    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7875a8-9210-4eda-8206-20ee8820c99a-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:00:55.907331 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.907250    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"
Nov 01 04:00:55.907331 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.907278    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85" event={"ID":"9d7875a8-9210-4eda-8206-20ee8820c99a","Type":"ContainerDied","Data":"ace9e7d89cd91c5ad76ba914bc5114fe9597ed6821ab375446375f089cb19f3c"}
Nov 01 04:00:55.907331 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:00:55.907326    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ace9e7d89cd91c5ad76ba914bc5114fe9597ed6821ab375446375f089cb19f3c"
Nov 01 04:00:58.724885 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:00:58.724851    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:01:05.849628 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:05.849553    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"]
Nov 01 04:01:05.866446 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:05.851715    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9d7875a8-9210-4eda-8206-20ee8820c99a" containerName="revoke-keys"
Nov 01 04:01:05.866690 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:05.866663    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7875a8-9210-4eda-8206-20ee8820c99a" containerName="revoke-keys"
Nov 01 04:01:05.867034 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:05.867015    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7875a8-9210-4eda-8206-20ee8820c99a" containerName="revoke-keys"
Nov 01 04:01:05.884274 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:05.884238    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:05.897992 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:05.894349    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"]
Nov 01 04:01:05.918289 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:05.918251    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-973dabb3-pxxkh\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:05.918588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:05.918567    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-973dabb3-pxxkh\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:05.918749 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:05.918733    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p7qt\" (UniqueName: \"kubernetes.io/projected/525a464b-f703-48aa-913e-015a3ccda41b-kube-api-access-6p7qt\") pod \"maas-api-revoke-sub-973dabb3-pxxkh\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:05.936701 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:05.936657    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:01:06.009511 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:06.007596    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:01:06.020792 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:06.020755    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-973dabb3-pxxkh\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:06.021105 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:06.021084    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-973dabb3-pxxkh\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:06.021220 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:06.021207    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p7qt\" (UniqueName: \"kubernetes.io/projected/525a464b-f703-48aa-913e-015a3ccda41b-kube-api-access-6p7qt\") pod \"maas-api-revoke-sub-973dabb3-pxxkh\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:06.022547 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:06.022507    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-973dabb3-pxxkh\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:06.025769 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:06.025746    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-973dabb3-pxxkh\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:06.087948 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:06.087910    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p7qt\" (UniqueName: \"kubernetes.io/projected/525a464b-f703-48aa-913e-015a3ccda41b-kube-api-access-6p7qt\") pod \"maas-api-revoke-sub-973dabb3-pxxkh\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:06.292177 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:06.289696    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:06.764007 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:06.763971    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"]
Nov 01 04:01:07.040363 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:07.040285    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh" event={"ID":"525a464b-f703-48aa-913e-015a3ccda41b","Type":"ContainerStarted","Data":"b96b3f4b4fb04880c255977f103d827ae0452c963f8744b48f9b7a4ce129b655"}
Nov 01 04:01:08.048343 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:08.048313    4882 generic.go:356] "Generic (PLEG): container finished" podID="525a464b-f703-48aa-913e-015a3ccda41b" containerID="e27b7dcba89b40db11096c128fd74f3447c91093f58de138875a7de0f7b2705c" exitCode=0
Nov 01 04:01:08.048709 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:08.048363    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh" event={"ID":"525a464b-f703-48aa-913e-015a3ccda41b","Type":"ContainerDied","Data":"e27b7dcba89b40db11096c128fd74f3447c91093f58de138875a7de0f7b2705c"}
Nov 01 04:01:09.193119 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:09.193095    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:09.220214 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:09.220187    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-xks-ca\") pod \"525a464b-f703-48aa-913e-015a3ccda41b\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") "
Nov 01 04:01:09.220346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:09.220232    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-openshift-ca\") pod \"525a464b-f703-48aa-913e-015a3ccda41b\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") "
Nov 01 04:01:09.220346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:09.220255    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525a464b-f703-48aa-913e-015a3ccda41b-kube-api-access-6p7qt\" (UniqueName: \"kubernetes.io/projected/525a464b-f703-48aa-913e-015a3ccda41b-kube-api-access-6p7qt\") pod \"525a464b-f703-48aa-913e-015a3ccda41b\" (UID: \"525a464b-f703-48aa-913e-015a3ccda41b\") "
Nov 01 04:01:09.220731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:09.220701    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-openshift-ca" pod "525a464b-f703-48aa-913e-015a3ccda41b" (UID: "525a464b-f703-48aa-913e-015a3ccda41b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:01:09.220865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:09.220820    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:09.222211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:09.222181    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-xks-ca" pod "525a464b-f703-48aa-913e-015a3ccda41b" (UID: "525a464b-f703-48aa-913e-015a3ccda41b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:01:09.222560 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:09.222540    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525a464b-f703-48aa-913e-015a3ccda41b-kube-api-access-6p7qt" pod "525a464b-f703-48aa-913e-015a3ccda41b" (UID: "525a464b-f703-48aa-913e-015a3ccda41b"). InnerVolumeSpecName "kube-api-access-6p7qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:01:09.322463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:09.322380    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525a464b-f703-48aa-913e-015a3ccda41b-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:09.322463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:09.322409    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p7qt\" (UniqueName: \"kubernetes.io/projected/525a464b-f703-48aa-913e-015a3ccda41b-kube-api-access-6p7qt\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:09.565269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:09.565237    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"]
Nov 01 04:01:10.063254 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:10.063225    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"
Nov 01 04:01:10.063465 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:10.063268    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh" event={"ID":"525a464b-f703-48aa-913e-015a3ccda41b","Type":"ContainerDied","Data":"b96b3f4b4fb04880c255977f103d827ae0452c963f8744b48f9b7a4ce129b655"}
Nov 01 04:01:10.063465 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:10.063319    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b96b3f4b4fb04880c255977f103d827ae0452c963f8744b48f9b7a4ce129b655"
Nov 01 04:01:15.444465 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.444419    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"]
Nov 01 04:01:15.444951 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.444938    4882 cpu_manager.go:414] "containerMap: removing container" podUID="525a464b-f703-48aa-913e-015a3ccda41b" containerName="revoke-keys"
Nov 01 04:01:15.444994 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.444951    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525a464b-f703-48aa-913e-015a3ccda41b" containerName="revoke-keys"
Nov 01 04:01:15.445056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.445047    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="525a464b-f703-48aa-913e-015a3ccda41b" containerName="revoke-keys"
Nov 01 04:01:15.449899 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.449879    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:15.489642 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.489610    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rztk\" (UniqueName: \"kubernetes.io/projected/f4a926b3-8d49-4068-b19b-72709e72aca2-kube-api-access-2rztk\") pod \"maas-api-revoke-sub-c6109dfd-d7x5h\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:15.489799 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.489675    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6109dfd-d7x5h\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:15.489842 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.489803    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6109dfd-d7x5h\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:15.502868 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.502697    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"]
Nov 01 04:01:15.544666 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.544637    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:01:15.544806 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.544666    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:01:15.591218 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.591183    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6109dfd-d7x5h\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:15.591382 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.591329    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rztk\" (UniqueName: \"kubernetes.io/projected/f4a926b3-8d49-4068-b19b-72709e72aca2-kube-api-access-2rztk\") pod \"maas-api-revoke-sub-c6109dfd-d7x5h\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:15.591465 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.591395    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6109dfd-d7x5h\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:15.591846 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.591821    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6109dfd-d7x5h\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:15.593666 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.593646    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6109dfd-d7x5h\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:15.648408 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.648377    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rztk\" (UniqueName: \"kubernetes.io/projected/f4a926b3-8d49-4068-b19b-72709e72aca2-kube-api-access-2rztk\") pod \"maas-api-revoke-sub-c6109dfd-d7x5h\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:15.763054 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:15.762982    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:16.133080 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:16.133052    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"]
Nov 01 04:01:16.134530 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:01:16.134502    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a926b3_8d49_4068_b19b_72709e72aca2.slice/crio-0a7b01b717c792f33447746328f9bc7b78bea2dc19d7bf64a5c363555d2b2c72 WatchSource:0}: Error finding container 0a7b01b717c792f33447746328f9bc7b78bea2dc19d7bf64a5c363555d2b2c72: Status 404 returned error can't find the container with id 0a7b01b717c792f33447746328f9bc7b78bea2dc19d7bf64a5c363555d2b2c72
Nov 01 04:01:17.110584 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:17.110548    4882 generic.go:356] "Generic (PLEG): container finished" podID="f4a926b3-8d49-4068-b19b-72709e72aca2" containerID="fddff4c83894f2e36df50e595ccb0a841a1cca48bdff6e24c403769a942b0bea" exitCode=0
Nov 01 04:01:17.111006 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:17.110612    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h" event={"ID":"f4a926b3-8d49-4068-b19b-72709e72aca2","Type":"ContainerDied","Data":"fddff4c83894f2e36df50e595ccb0a841a1cca48bdff6e24c403769a942b0bea"}
Nov 01 04:01:17.111006 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:17.110654    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h" event={"ID":"f4a926b3-8d49-4068-b19b-72709e72aca2","Type":"ContainerStarted","Data":"0a7b01b717c792f33447746328f9bc7b78bea2dc19d7bf64a5c363555d2b2c72"}
Nov 01 04:01:18.285955 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:18.285930    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:18.424661 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:18.424559    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a926b3-8d49-4068-b19b-72709e72aca2-kube-api-access-2rztk\" (UniqueName: \"kubernetes.io/projected/f4a926b3-8d49-4068-b19b-72709e72aca2-kube-api-access-2rztk\") pod \"f4a926b3-8d49-4068-b19b-72709e72aca2\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") "
Nov 01 04:01:18.424661 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:18.424616    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-xks-ca\") pod \"f4a926b3-8d49-4068-b19b-72709e72aca2\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") "
Nov 01 04:01:18.424889 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:18.424779    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-openshift-ca\") pod \"f4a926b3-8d49-4068-b19b-72709e72aca2\" (UID: \"f4a926b3-8d49-4068-b19b-72709e72aca2\") "
Nov 01 04:01:18.425194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:18.425161    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-openshift-ca" pod "f4a926b3-8d49-4068-b19b-72709e72aca2" (UID: "f4a926b3-8d49-4068-b19b-72709e72aca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:01:18.425404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:18.425381    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:18.427332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:18.427302    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a926b3-8d49-4068-b19b-72709e72aca2-kube-api-access-2rztk" pod "f4a926b3-8d49-4068-b19b-72709e72aca2" (UID: "f4a926b3-8d49-4068-b19b-72709e72aca2"). InnerVolumeSpecName "kube-api-access-2rztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:01:18.428135 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:18.428114    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-xks-ca" pod "f4a926b3-8d49-4068-b19b-72709e72aca2" (UID: "f4a926b3-8d49-4068-b19b-72709e72aca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:01:18.526533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:18.526503    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rztk\" (UniqueName: \"kubernetes.io/projected/f4a926b3-8d49-4068-b19b-72709e72aca2-kube-api-access-2rztk\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:18.526533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:18.526529    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a926b3-8d49-4068-b19b-72709e72aca2-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:19.123206 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:19.123168    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"
Nov 01 04:01:19.123206 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:19.123199    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h" event={"ID":"f4a926b3-8d49-4068-b19b-72709e72aca2","Type":"ContainerDied","Data":"0a7b01b717c792f33447746328f9bc7b78bea2dc19d7bf64a5c363555d2b2c72"}
Nov 01 04:01:19.123418 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:19.123250    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a7b01b717c792f33447746328f9bc7b78bea2dc19d7bf64a5c363555d2b2c72"
Nov 01 04:01:24.586084 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:24.586032    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz" podUID="03db033d-d120-4ae6-8610-b2ebad7510dd" containerName="main" containerID="cri-o://cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6" gracePeriod=45
Nov 01 04:01:24.860952 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:24.860925    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 04:01:25.012923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.012887    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-model-cache\") pod \"03db033d-d120-4ae6-8610-b2ebad7510dd\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") "
Nov 01 04:01:25.013096 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.012977    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-tmp-dir\") pod \"03db033d-d120-4ae6-8610-b2ebad7510dd\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") "
Nov 01 04:01:25.013096 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.013035    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-dshm\") pod \"03db033d-d120-4ae6-8610-b2ebad7510dd\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") "
Nov 01 04:01:25.013229 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.013202    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-model-cache" pod "03db033d-d120-4ae6-8610-b2ebad7510dd" (UID: "03db033d-d120-4ae6-8610-b2ebad7510dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:01:25.013229 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.013219    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-tmp-dir" pod "03db033d-d120-4ae6-8610-b2ebad7510dd" (UID: "03db033d-d120-4ae6-8610-b2ebad7510dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:01:25.013329 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.013253    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03db033d-d120-4ae6-8610-b2ebad7510dd-kube-api-access-gfvct\" (UniqueName: \"kubernetes.io/projected/03db033d-d120-4ae6-8610-b2ebad7510dd-kube-api-access-gfvct\") pod \"03db033d-d120-4ae6-8610-b2ebad7510dd\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") "
Nov 01 04:01:25.013329 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.013287    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03db033d-d120-4ae6-8610-b2ebad7510dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/03db033d-d120-4ae6-8610-b2ebad7510dd-tls-certs\") pod \"03db033d-d120-4ae6-8610-b2ebad7510dd\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") "
Nov 01 04:01:25.013454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.013337    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-home\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-home\") pod \"03db033d-d120-4ae6-8610-b2ebad7510dd\" (UID: \"03db033d-d120-4ae6-8610-b2ebad7510dd\") "
Nov 01 04:01:25.013695 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.013672    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-home" pod "03db033d-d120-4ae6-8610-b2ebad7510dd" (UID: "03db033d-d120-4ae6-8610-b2ebad7510dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:01:25.013923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.013864    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-home\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:25.013923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.013883    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-model-cache\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:25.013923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.013896    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-tmp-dir\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:25.015523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.015501    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03db033d-d120-4ae6-8610-b2ebad7510dd-kube-api-access-gfvct" pod "03db033d-d120-4ae6-8610-b2ebad7510dd" (UID: "03db033d-d120-4ae6-8610-b2ebad7510dd"). InnerVolumeSpecName "kube-api-access-gfvct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:01:25.015660 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.015638    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-dshm" pod "03db033d-d120-4ae6-8610-b2ebad7510dd" (UID: "03db033d-d120-4ae6-8610-b2ebad7510dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:01:25.015776 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.015716    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03db033d-d120-4ae6-8610-b2ebad7510dd-tls-certs" pod "03db033d-d120-4ae6-8610-b2ebad7510dd" (UID: "03db033d-d120-4ae6-8610-b2ebad7510dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:01:25.114909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.114829    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfvct\" (UniqueName: \"kubernetes.io/projected/03db033d-d120-4ae6-8610-b2ebad7510dd-kube-api-access-gfvct\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:25.114909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.114863    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03db033d-d120-4ae6-8610-b2ebad7510dd-tls-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:25.114909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.114873    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03db033d-d120-4ae6-8610-b2ebad7510dd-dshm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:25.174438 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.167771    4882 generic.go:356] "Generic (PLEG): container finished" podID="03db033d-d120-4ae6-8610-b2ebad7510dd" containerID="cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6" exitCode=0
Nov 01 04:01:25.174438 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.167995    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz" event={"ID":"03db033d-d120-4ae6-8610-b2ebad7510dd","Type":"ContainerDied","Data":"cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6"}
Nov 01 04:01:25.174438 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.168035    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz" event={"ID":"03db033d-d120-4ae6-8610-b2ebad7510dd","Type":"ContainerDied","Data":"425dfc3fdc7d9dcc5994d6ab861b838b4783cb140a2f6aed21003c00d568f90e"}
Nov 01 04:01:25.174438 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.168058    4882 scope.go:122] "RemoveContainer" containerID="cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6"
Nov 01 04:01:25.174438 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.168220    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"
Nov 01 04:01:25.268984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.268958    4882 scope.go:122] "RemoveContainer" containerID="cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6"
Nov 01 04:01:25.270980 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:01:25.270955    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6\": container with ID starting with cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6 not found: ID does not exist" containerID="cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6"
Nov 01 04:01:25.271219 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.271187    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6"} err="failed to get container status \"cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6\": rpc error: code = NotFound desc = could not find container \"cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6\": container with ID starting with cb1060d167d7b08213e2cbc1cc4c784f623c0080cca9bb2c4af3f42abe2bdfc6 not found: ID does not exist"
Nov 01 04:01:25.362456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.353532    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"]
Nov 01 04:01:25.390506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.383925    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-a-9e9797da-kserve-6cd568d6bd-bflhz"]
Nov 01 04:01:25.689966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:25.689932    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03db033d-d120-4ae6-8610-b2ebad7510dd" path="/var/lib/kubelet/pods/03db033d-d120-4ae6-8610-b2ebad7510dd/volumes"
Nov 01 04:01:26.612453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:26.607543    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:26.627556 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:26.627506    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52" podUID="91fcd874-e74a-4bf5-a6ca-3842296d22b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:26.668453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:26.664501    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Nov 01 04:01:26.682450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:26.681506    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:26.895708 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:26.895588    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:26.944703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:26.944641    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:27.156510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.155522    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:01:27.156510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.155591    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:27.208404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.208331    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:27.361451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.359532    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 04:01:27.652675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.650506    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:27.652675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.651160    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 04:01:27.697452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.692522    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52" podUID="91fcd874-e74a-4bf5-a6ca-3842296d22b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:27.697452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.692915    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 04:01:27.697452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.693574    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:27.697452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.693706    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 04:01:27.937532 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.928543    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:27.937532 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.928894    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Nov 01 04:01:27.965453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.962685    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:01:28.042476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:27.978509    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:28.042476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:28.006680    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 04:01:28.211911 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:28.211748    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:01:28.212346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:28.212311    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 04:01:28.313966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:28.313933    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 04:01:28.361455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:28.361029    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 04:01:28.361455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:28.361295    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 04:01:28.362110 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:28.362088    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:01:28.386243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:28.386217    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 04:01:28.456585 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:28.455231    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 04:01:28.739471 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:01:28.739393    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:01:30.958492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:30.958458    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"]
Nov 01 04:01:30.958986 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:30.958973    4882 cpu_manager.go:414] "containerMap: removing container" podUID="03db033d-d120-4ae6-8610-b2ebad7510dd" containerName="main"
Nov 01 04:01:30.959044 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:30.958986    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03db033d-d120-4ae6-8610-b2ebad7510dd" containerName="main"
Nov 01 04:01:30.959044 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:30.959013    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f4a926b3-8d49-4068-b19b-72709e72aca2" containerName="revoke-keys"
Nov 01 04:01:30.959044 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:30.959018    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a926b3-8d49-4068-b19b-72709e72aca2" containerName="revoke-keys"
Nov 01 04:01:30.959142 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:30.959098    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a926b3-8d49-4068-b19b-72709e72aca2" containerName="revoke-keys"
Nov 01 04:01:30.959142 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:30.959112    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="03db033d-d120-4ae6-8610-b2ebad7510dd" containerName="main"
Nov 01 04:01:30.962593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:30.962574    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:30.966075 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:30.966044    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58600c40-b54tr\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:30.966214 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:30.966196    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58600c40-b54tr\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:30.966291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:30.966226    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm5pg\" (UniqueName: \"kubernetes.io/projected/2127a957-f579-46ce-befe-2437011ddfe9-kube-api-access-pm5pg\") pod \"maas-api-revoke-sub-58600c40-b54tr\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:31.058235 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.058204    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:01:31.058235 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.058212    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:01:31.064047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.064018    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"]
Nov 01 04:01:31.067339 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.067316    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58600c40-b54tr\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:31.067476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.067348    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm5pg\" (UniqueName: \"kubernetes.io/projected/2127a957-f579-46ce-befe-2437011ddfe9-kube-api-access-pm5pg\") pod \"maas-api-revoke-sub-58600c40-b54tr\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:31.067476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.067397    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58600c40-b54tr\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:31.068025 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.068000    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58600c40-b54tr\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:31.069861 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.069838    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58600c40-b54tr\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:31.124645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.124616    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm5pg\" (UniqueName: \"kubernetes.io/projected/2127a957-f579-46ce-befe-2437011ddfe9-kube-api-access-pm5pg\") pod \"maas-api-revoke-sub-58600c40-b54tr\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:31.277244 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.277158    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:31.650261 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.650234    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"]
Nov 01 04:01:31.652645 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:01:31.652620    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2127a957_f579_46ce_befe_2437011ddfe9.slice/crio-df33f555ed938d69f845b30664dc855d2f25669b6cc079e74c7ae1ab254e189b WatchSource:0}: Error finding container df33f555ed938d69f845b30664dc855d2f25669b6cc079e74c7ae1ab254e189b: Status 404 returned error can't find the container with id df33f555ed938d69f845b30664dc855d2f25669b6cc079e74c7ae1ab254e189b
Nov 01 04:01:31.734098 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.734070    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"]
Nov 01 04:01:31.739402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.739383    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:31.784083 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.784059    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8611e76-ccmq7\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:31.784261 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.784243    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8611e76-ccmq7\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:31.784309 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.784298    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzh2\" (UniqueName: \"kubernetes.io/projected/e5193c36-42c8-46da-9d1c-ff27a0799492-kube-api-access-vlzh2\") pod \"maas-api-revoke-sub-f8611e76-ccmq7\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:31.793732 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.793707    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"]
Nov 01 04:01:31.885693 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.885667    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8611e76-ccmq7\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:31.885860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.885841    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8611e76-ccmq7\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:31.885930 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.885914    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzh2\" (UniqueName: \"kubernetes.io/projected/e5193c36-42c8-46da-9d1c-ff27a0799492-kube-api-access-vlzh2\") pod \"maas-api-revoke-sub-f8611e76-ccmq7\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:31.886546 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.886522    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8611e76-ccmq7\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:31.887944 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.887926    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8611e76-ccmq7\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:31.942954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:31.942927    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzh2\" (UniqueName: \"kubernetes.io/projected/e5193c36-42c8-46da-9d1c-ff27a0799492-kube-api-access-vlzh2\") pod \"maas-api-revoke-sub-f8611e76-ccmq7\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:32.051984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.051956    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:32.292683 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.292601    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr" event={"ID":"2127a957-f579-46ce-befe-2437011ddfe9","Type":"ContainerStarted","Data":"45bd7dcf219981af41c29c971309b16994f6e6d0c1e6328d20686c2c36b0bde8"}
Nov 01 04:01:32.292683 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.292651    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr" event={"ID":"2127a957-f579-46ce-befe-2437011ddfe9","Type":"ContainerStarted","Data":"df33f555ed938d69f845b30664dc855d2f25669b6cc079e74c7ae1ab254e189b"}
Nov 01 04:01:32.402935 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.402883    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr" podStartSLOduration=2.402869844 podStartE2EDuration="2.402869844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:32.399574274 +0000 UTC m=+1160.961607299" watchObservedRunningTime="2026-11-01 04:01:32.402869844 +0000 UTC m=+1160.964902861"
Nov 01 04:01:32.427917 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:01:32.427885    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5193c36_42c8_46da_9d1c_ff27a0799492.slice/crio-437bf25b294e67bf6b25cd63d3b5a42570caef97288149052dac63eb03806895 WatchSource:0}: Error finding container 437bf25b294e67bf6b25cd63d3b5a42570caef97288149052dac63eb03806895: Status 404 returned error can't find the container with id 437bf25b294e67bf6b25cd63d3b5a42570caef97288149052dac63eb03806895
Nov 01 04:01:32.449202 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.449176    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"]
Nov 01 04:01:32.713848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.713816    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"]
Nov 01 04:01:32.717813 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.717787    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.796875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.796381    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.796875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.796468    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.796875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.796506    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.796875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.796647    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.796875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.796698    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3aab38-b02f-45af-8784-33dd8a2522c2-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.796875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.796726    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.796875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.796784    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6nd\" (UniqueName: \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-kube-api-access-sb6nd\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.796875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.796842    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.796875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.796880    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.798544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.798517    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"]
Nov 01 04:01:32.898170 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898092    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.898170 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898138    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3aab38-b02f-45af-8784-33dd8a2522c2-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.898356 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898258    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.898356 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898335    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6nd\" (UniqueName: \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-kube-api-access-sb6nd\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.898480 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898404    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.898480 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898470    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.898595 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898507    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.898595 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898564    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.898711 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898600    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.898711 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898683    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.898812 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898753    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.899127 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898837    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.899127 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.898914    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3aab38-b02f-45af-8784-33dd8a2522c2-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.899127 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.899071    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.900761 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.900740    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.900913 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.900896    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.955278 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.955243    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:32.955411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:32.955345    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6nd\" (UniqueName: \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-kube-api-access-sb6nd\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:33.030801 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:33.030765    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:33.300209 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:33.300180    4882 generic.go:356] "Generic (PLEG): container finished" podID="2127a957-f579-46ce-befe-2437011ddfe9" containerID="45bd7dcf219981af41c29c971309b16994f6e6d0c1e6328d20686c2c36b0bde8" exitCode=0
Nov 01 04:01:33.300650 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:33.300254    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr" event={"ID":"2127a957-f579-46ce-befe-2437011ddfe9","Type":"ContainerDied","Data":"45bd7dcf219981af41c29c971309b16994f6e6d0c1e6328d20686c2c36b0bde8"}
Nov 01 04:01:33.301856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:33.301833    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7" event={"ID":"e5193c36-42c8-46da-9d1c-ff27a0799492","Type":"ContainerStarted","Data":"d53aab3937a722fa4d4a9d9d0eae5fac2f5f52f30d2633c86b550e0ba399f6d0"}
Nov 01 04:01:33.301973 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:33.301881    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7" event={"ID":"e5193c36-42c8-46da-9d1c-ff27a0799492","Type":"ContainerStarted","Data":"437bf25b294e67bf6b25cd63d3b5a42570caef97288149052dac63eb03806895"}
Nov 01 04:01:33.457533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:33.457502    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"]
Nov 01 04:01:33.460328 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:33.460291    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:01:33.460475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:33.460371    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:01:33.460475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:33.460462    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:01:33.517101 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:33.517048    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7" podStartSLOduration=2.517030568 podStartE2EDuration="2.517030568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:33.515747746 +0000 UTC m=+1162.077780763" watchObservedRunningTime="2026-11-01 04:01:33.517030568 +0000 UTC m=+1162.079063587"
Nov 01 04:01:34.149016 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.148971    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"]
Nov 01 04:01:34.155517 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.155489    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.213877 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.213844    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2qtg\" (UniqueName: \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-kube-api-access-g2qtg\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.214042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.213887    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.214042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.213904    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.214118 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.214071    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.214118 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.214104    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.214250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.214224    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.214285 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.214267    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551c4d53-6477-4592-99c4-486fd57ba643-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.214318 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.214293    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551c4d53-6477-4592-99c4-486fd57ba643-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.214352 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.214338    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.244459 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.244397    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"]
Nov 01 04:01:34.312544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.312511    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" event={"ID":"da3aab38-b02f-45af-8784-33dd8a2522c2","Type":"ContainerStarted","Data":"7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a"}
Nov 01 04:01:34.312954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.312563    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" event={"ID":"da3aab38-b02f-45af-8784-33dd8a2522c2","Type":"ContainerStarted","Data":"91e66683f6512db1a60df414ccb4181011a98c3ce7a1afaff2aadd1a2214aea4"}
Nov 01 04:01:34.314020 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.313991    4882 generic.go:356] "Generic (PLEG): container finished" podID="e5193c36-42c8-46da-9d1c-ff27a0799492" containerID="d53aab3937a722fa4d4a9d9d0eae5fac2f5f52f30d2633c86b550e0ba399f6d0" exitCode=0
Nov 01 04:01:34.314125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.314034    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7" event={"ID":"e5193c36-42c8-46da-9d1c-ff27a0799492","Type":"ContainerDied","Data":"d53aab3937a722fa4d4a9d9d0eae5fac2f5f52f30d2633c86b550e0ba399f6d0"}
Nov 01 04:01:34.315972 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.315948    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.316059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.315984    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551c4d53-6477-4592-99c4-486fd57ba643-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.316059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.316008    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551c4d53-6477-4592-99c4-486fd57ba643-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.316059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.316039    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.316218 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.316182    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2qtg\" (UniqueName: \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-kube-api-access-g2qtg\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.316271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.316218    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.316271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.316247    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.316362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.316338    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.316416 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.316367    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.317731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.316695    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.317731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.316797    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.317731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.317125    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551c4d53-6477-4592-99c4-486fd57ba643-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.317731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.317526    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.317731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.317697    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.318651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.318628    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.319272 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.319251    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551c4d53-6477-4592-99c4-486fd57ba643-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.395119 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.394997    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2qtg\" (UniqueName: \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-kube-api-access-g2qtg\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.395521 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.395493    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.455889 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.455826    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" podStartSLOduration=2.455808284 podStartE2EDuration="2.455808284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:34.451108521 +0000 UTC m=+1163.013141538" watchObservedRunningTime="2026-11-01 04:01:34.455808284 +0000 UTC m=+1163.017841302"
Nov 01 04:01:34.493189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.493162    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:34.511566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.511540    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:34.621177 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.619417    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-xks-ca\") pod \"2127a957-f579-46ce-befe-2437011ddfe9\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") "
Nov 01 04:01:34.621177 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.619662    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2127a957-f579-46ce-befe-2437011ddfe9-kube-api-access-pm5pg\" (UniqueName: \"kubernetes.io/projected/2127a957-f579-46ce-befe-2437011ddfe9-kube-api-access-pm5pg\") pod \"2127a957-f579-46ce-befe-2437011ddfe9\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") "
Nov 01 04:01:34.621177 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.620127    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-openshift-ca\") pod \"2127a957-f579-46ce-befe-2437011ddfe9\" (UID: \"2127a957-f579-46ce-befe-2437011ddfe9\") "
Nov 01 04:01:34.621177 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.620542    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-openshift-ca" pod "2127a957-f579-46ce-befe-2437011ddfe9" (UID: "2127a957-f579-46ce-befe-2437011ddfe9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:01:34.621743 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.621724    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-xks-ca" pod "2127a957-f579-46ce-befe-2437011ddfe9" (UID: "2127a957-f579-46ce-befe-2437011ddfe9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:01:34.623217 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.623180    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2127a957-f579-46ce-befe-2437011ddfe9-kube-api-access-pm5pg" pod "2127a957-f579-46ce-befe-2437011ddfe9" (UID: "2127a957-f579-46ce-befe-2437011ddfe9"). InnerVolumeSpecName "kube-api-access-pm5pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:01:34.721797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.721751    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm5pg\" (UniqueName: \"kubernetes.io/projected/2127a957-f579-46ce-befe-2437011ddfe9-kube-api-access-pm5pg\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:34.721797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.721785    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:34.721998 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.721807    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2127a957-f579-46ce-befe-2437011ddfe9-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:34.922681 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.922645    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"]
Nov 01 04:01:34.924809 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:01:34.924781    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551c4d53_6477_4592_99c4_486fd57ba643.slice/crio-82459156427dfb23d6a9cd28d5fdd829153c4cc6bd2151a34a4b898336b4852d WatchSource:0}: Error finding container 82459156427dfb23d6a9cd28d5fdd829153c4cc6bd2151a34a4b898336b4852d: Status 404 returned error can't find the container with id 82459156427dfb23d6a9cd28d5fdd829153c4cc6bd2151a34a4b898336b4852d
Nov 01 04:01:34.927028 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.926998    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:01:34.927106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.927061    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:01:34.927106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:34.927088    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:01:35.031169 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.031141    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:35.032409 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.032388    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 04:01:35.032511 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.032451    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 04:01:35.324539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.324461    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"
Nov 01 04:01:35.324539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.324478    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr" event={"ID":"2127a957-f579-46ce-befe-2437011ddfe9","Type":"ContainerDied","Data":"df33f555ed938d69f845b30664dc855d2f25669b6cc079e74c7ae1ab254e189b"}
Nov 01 04:01:35.325021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.324538    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df33f555ed938d69f845b30664dc855d2f25669b6cc079e74c7ae1ab254e189b"
Nov 01 04:01:35.326304 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.326231    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" event={"ID":"551c4d53-6477-4592-99c4-486fd57ba643","Type":"ContainerStarted","Data":"ae3f108bd6c8da76564c970a948cf1511e73e74a80282f4f0cf44d57dd7bd295"}
Nov 01 04:01:35.326304 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.326275    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" event={"ID":"551c4d53-6477-4592-99c4-486fd57ba643","Type":"ContainerStarted","Data":"82459156427dfb23d6a9cd28d5fdd829153c4cc6bd2151a34a4b898336b4852d"}
Nov 01 04:01:35.459279 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.453463    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" podStartSLOduration=1.453443863 podStartE2EDuration="1.453443863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:35.451636649 +0000 UTC m=+1164.013669669" watchObservedRunningTime="2026-11-01 04:01:35.453443863 +0000 UTC m=+1164.015476877"
Nov 01 04:01:35.493640 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.493593    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:35.546611 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.546484    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:35.637896 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.637818    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-openshift-ca\") pod \"e5193c36-42c8-46da-9d1c-ff27a0799492\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") "
Nov 01 04:01:35.638071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.637947    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-xks-ca\") pod \"e5193c36-42c8-46da-9d1c-ff27a0799492\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") "
Nov 01 04:01:35.638071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.638005    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5193c36-42c8-46da-9d1c-ff27a0799492-kube-api-access-vlzh2\" (UniqueName: \"kubernetes.io/projected/e5193c36-42c8-46da-9d1c-ff27a0799492-kube-api-access-vlzh2\") pod \"e5193c36-42c8-46da-9d1c-ff27a0799492\" (UID: \"e5193c36-42c8-46da-9d1c-ff27a0799492\") "
Nov 01 04:01:35.638193 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.638141    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-openshift-ca" pod "e5193c36-42c8-46da-9d1c-ff27a0799492" (UID: "e5193c36-42c8-46da-9d1c-ff27a0799492"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:01:35.638313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.638297    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:35.640047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.640023    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-xks-ca" pod "e5193c36-42c8-46da-9d1c-ff27a0799492" (UID: "e5193c36-42c8-46da-9d1c-ff27a0799492"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:01:35.640752 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.640735    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5193c36-42c8-46da-9d1c-ff27a0799492-kube-api-access-vlzh2" pod "e5193c36-42c8-46da-9d1c-ff27a0799492" (UID: "e5193c36-42c8-46da-9d1c-ff27a0799492"). InnerVolumeSpecName "kube-api-access-vlzh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:01:35.739126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.739095    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5193c36-42c8-46da-9d1c-ff27a0799492-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:35.739126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:35.739123    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlzh2\" (UniqueName: \"kubernetes.io/projected/e5193c36-42c8-46da-9d1c-ff27a0799492-kube-api-access-vlzh2\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:36.031498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:36.031421    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 04:01:36.031718 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:36.031515    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 04:01:36.343524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:36.343405    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7" event={"ID":"e5193c36-42c8-46da-9d1c-ff27a0799492","Type":"ContainerDied","Data":"437bf25b294e67bf6b25cd63d3b5a42570caef97288149052dac63eb03806895"}
Nov 01 04:01:36.343524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:36.343469    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"
Nov 01 04:01:36.343930 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:36.343529    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="437bf25b294e67bf6b25cd63d3b5a42570caef97288149052dac63eb03806895"
Nov 01 04:01:36.511640 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:36.511550    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 04:01:36.511640 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:36.511623    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" podUID="551c4d53-6477-4592-99c4-486fd57ba643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 04:01:37.031208 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:37.031164    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 04:01:37.031382 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:37.031216    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 04:01:37.501481 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:37.500836    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 04:01:37.501481 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:37.500897    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" podUID="551c4d53-6477-4592-99c4-486fd57ba643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 04:01:38.031911 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:38.031868    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 04:01:38.032118 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:38.031934    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 04:01:38.499454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:38.493860    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 04:01:38.499454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:38.493921    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" podUID="551c4d53-6477-4592-99c4-486fd57ba643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 04:01:39.031799 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:39.031761    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 04:01:39.032278 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:39.031819    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 04:01:39.502451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:39.501827    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 04:01:39.502451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:39.501888    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" podUID="551c4d53-6477-4592-99c4-486fd57ba643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 04:01:40.031734 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:40.031696    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 04:01:40.031912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:40.031749    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 04:01:40.494023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:40.493983    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 04:01:40.494255 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:40.494064    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" podUID="551c4d53-6477-4592-99c4-486fd57ba643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 04:01:41.113216 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.113175    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:41.114565 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.114529    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:41.115216 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.115201    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:01:41.493765 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.493729    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 04:01:41.493951 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.493785    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" podUID="551c4d53-6477-4592-99c4-486fd57ba643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 04:01:41.721100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.721065    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"]
Nov 01 04:01:41.721621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.721606    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2127a957-f579-46ce-befe-2437011ddfe9" containerName="revoke-keys"
Nov 01 04:01:41.721677 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.721623    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2127a957-f579-46ce-befe-2437011ddfe9" containerName="revoke-keys"
Nov 01 04:01:41.721677 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.721641    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e5193c36-42c8-46da-9d1c-ff27a0799492" containerName="revoke-keys"
Nov 01 04:01:41.721677 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.721651    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5193c36-42c8-46da-9d1c-ff27a0799492" containerName="revoke-keys"
Nov 01 04:01:41.721776 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.721749    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5193c36-42c8-46da-9d1c-ff27a0799492" containerName="revoke-keys"
Nov 01 04:01:41.721776 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.721767    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2127a957-f579-46ce-befe-2437011ddfe9" containerName="revoke-keys"
Nov 01 04:01:41.726629 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.726609    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:41.776404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.776286    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:01:41.776404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.776290    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:01:41.780443 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.780403    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"]
Nov 01 04:01:41.856754 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.856719    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284101ce-dqvvg\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:41.856924 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.856827    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsgv\" (UniqueName: \"kubernetes.io/projected/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-kube-api-access-zhsgv\") pod \"maas-api-revoke-sub-284101ce-dqvvg\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:41.856924 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.856871    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284101ce-dqvvg\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:41.958042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.958010    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284101ce-dqvvg\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:41.958249 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.958075    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsgv\" (UniqueName: \"kubernetes.io/projected/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-kube-api-access-zhsgv\") pod \"maas-api-revoke-sub-284101ce-dqvvg\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:41.958249 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.958126    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284101ce-dqvvg\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:41.958759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.958726    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-284101ce-dqvvg\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:41.960479 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:41.960455    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-284101ce-dqvvg\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:42.015357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:42.015331    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsgv\" (UniqueName: \"kubernetes.io/projected/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-kube-api-access-zhsgv\") pod \"maas-api-revoke-sub-284101ce-dqvvg\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:42.040446 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:42.040372    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:42.420391 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:42.420364    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"]
Nov 01 04:01:42.422255 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:01:42.422224    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eeb7e4f_1b49_41d6_aa6c_8387544bfc11.slice/crio-32be5047c31e20f8ec7eb90fef74e25a6f54f99213b43e2ae2dc3dc8c12f87a1 WatchSource:0}: Error finding container 32be5047c31e20f8ec7eb90fef74e25a6f54f99213b43e2ae2dc3dc8c12f87a1: Status 404 returned error can't find the container with id 32be5047c31e20f8ec7eb90fef74e25a6f54f99213b43e2ae2dc3dc8c12f87a1
Nov 01 04:01:42.498495 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:42.498470    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:42.498949 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:42.498929    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:42.499539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:42.499518    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:01:43.424605 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:43.424568    4882 generic.go:356] "Generic (PLEG): container finished" podID="5eeb7e4f-1b49-41d6-aa6c-8387544bfc11" containerID="b114e3d02140546ca7fe750ed991a57598e6c0f8e6487b14aac50b4756c14648" exitCode=0
Nov 01 04:01:43.425085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:43.424651    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg" event={"ID":"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11","Type":"ContainerDied","Data":"b114e3d02140546ca7fe750ed991a57598e6c0f8e6487b14aac50b4756c14648"}
Nov 01 04:01:43.425085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:43.424699    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg" event={"ID":"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11","Type":"ContainerStarted","Data":"32be5047c31e20f8ec7eb90fef74e25a6f54f99213b43e2ae2dc3dc8c12f87a1"}
Nov 01 04:01:44.563142 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.563113    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"]
Nov 01 04:01:44.567345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.567321    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.590340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.590312    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:44.707243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.706740    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-kube-api-access-zhsgv\" (UniqueName: \"kubernetes.io/projected/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-kube-api-access-zhsgv\") pod \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") "
Nov 01 04:01:44.707976 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.707583    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-xks-ca\") pod \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") "
Nov 01 04:01:44.708294 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.708273    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-openshift-ca\") pod \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\" (UID: \"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11\") "
Nov 01 04:01:44.709022 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.708898    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-openshift-ca" pod "5eeb7e4f-1b49-41d6-aa6c-8387544bfc11" (UID: "5eeb7e4f-1b49-41d6-aa6c-8387544bfc11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:01:44.731530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.711632    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.731922 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.731882    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.737757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.732235    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fbm8\" (UniqueName: \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-kube-api-access-9fbm8\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.738553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.738134    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.738775 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.738738    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.739023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.739005    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9975f6aa-7802-46fc-9612-fd00544fa547-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.739235 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.739217    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.739371 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.739356    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9975f6aa-7802-46fc-9612-fd00544fa547-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.743570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.743542    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.816631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.816596    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:44.816631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.815820    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"]
Nov 01 04:01:44.816873 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.783215    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-xks-ca" pod "5eeb7e4f-1b49-41d6-aa6c-8387544bfc11" (UID: "5eeb7e4f-1b49-41d6-aa6c-8387544bfc11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:01:44.824782 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.824746    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-kube-api-access-zhsgv" pod "5eeb7e4f-1b49-41d6-aa6c-8387544bfc11" (UID: "5eeb7e4f-1b49-41d6-aa6c-8387544bfc11"). InnerVolumeSpecName "kube-api-access-zhsgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:01:44.917815 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.917754    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.917997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.917840    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9975f6aa-7802-46fc-9612-fd00544fa547-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.917997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.917897    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.917997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.917940    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.918144 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.918003    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.918144 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.918090    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fbm8\" (UniqueName: \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-kube-api-access-9fbm8\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.918144 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.918127    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.918294 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.918156    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.918294 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.918240    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9975f6aa-7802-46fc-9612-fd00544fa547-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.918395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.918300    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.918395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.918327    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.918395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.918341    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsgv\" (UniqueName: \"kubernetes.io/projected/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-kube-api-access-zhsgv\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:44.918575 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.918448    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:44.918707 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.918636    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.918976 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.918956    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.919167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.919143    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9975f6aa-7802-46fc-9612-fd00544fa547-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.920489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.920470    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.920864 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.920849    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9975f6aa-7802-46fc-9612-fd00544fa547-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.977317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.977251    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:44.977317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:44.977291    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fbm8\" (UniqueName: \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-kube-api-access-9fbm8\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:45.199701 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:45.199668    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:45.439748 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:45.439713    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"
Nov 01 04:01:45.439919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:45.439746    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg" event={"ID":"5eeb7e4f-1b49-41d6-aa6c-8387544bfc11","Type":"ContainerDied","Data":"32be5047c31e20f8ec7eb90fef74e25a6f54f99213b43e2ae2dc3dc8c12f87a1"}
Nov 01 04:01:45.439919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:45.439806    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32be5047c31e20f8ec7eb90fef74e25a6f54f99213b43e2ae2dc3dc8c12f87a1"
Nov 01 04:01:45.642935 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:45.642886    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"]
Nov 01 04:01:45.647510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:45.647481    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:01:45.647606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:45.647561    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:01:45.647649 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:45.647602    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:01:46.448582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:46.448545    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" event={"ID":"9975f6aa-7802-46fc-9612-fd00544fa547","Type":"ContainerStarted","Data":"b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1"}
Nov 01 04:01:46.448763 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:46.448605    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" event={"ID":"9975f6aa-7802-46fc-9612-fd00544fa547","Type":"ContainerStarted","Data":"dd6c4601e9f4e6bc43aeaebfa1578c8621df35929a5118f1fb63fb7dd4e9db9b"}
Nov 01 04:01:46.588530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:46.588470    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" podStartSLOduration=2.588453601 podStartE2EDuration="2.588453601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:46.586676403 +0000 UTC m=+1175.148709421" watchObservedRunningTime="2026-11-01 04:01:46.588453601 +0000 UTC m=+1175.150486615"
Nov 01 04:01:47.200511 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:47.200465    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:47.202294 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:47.202265    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 04:01:47.202459 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:47.202332    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 04:01:48.201006 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.200958    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 04:01:48.201470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.201028    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 04:01:48.492818 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.492737    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"]
Nov 01 04:01:48.493264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.493244    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5eeb7e4f-1b49-41d6-aa6c-8387544bfc11" containerName="revoke-keys"
Nov 01 04:01:48.493264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.493259    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eeb7e4f-1b49-41d6-aa6c-8387544bfc11" containerName="revoke-keys"
Nov 01 04:01:48.493448 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.493382    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eeb7e4f-1b49-41d6-aa6c-8387544bfc11" containerName="revoke-keys"
Nov 01 04:01:48.497160 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.497135    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:48.528074 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.528050    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"]
Nov 01 04:01:48.545880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.545843    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:01:48.586748 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.586714    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:01:48.596749 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.596708    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc8c1d0-rmfkv\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:48.596939 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.596915    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc8c1d0-rmfkv\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:48.597076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.597054    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d822n\" (UniqueName: \"kubernetes.io/projected/6db616db-f37c-4b49-84c6-4f3684d6a295-kube-api-access-d822n\") pod \"maas-api-revoke-sub-bfc8c1d0-rmfkv\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:48.698807 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.698773    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc8c1d0-rmfkv\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:48.698968 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.698917    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc8c1d0-rmfkv\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:48.699010 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.698994    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d822n\" (UniqueName: \"kubernetes.io/projected/6db616db-f37c-4b49-84c6-4f3684d6a295-kube-api-access-d822n\") pod \"maas-api-revoke-sub-bfc8c1d0-rmfkv\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:48.699512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.699491    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc8c1d0-rmfkv\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:48.701370 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.701344    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc8c1d0-rmfkv\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:48.756197 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.756133    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d822n\" (UniqueName: \"kubernetes.io/projected/6db616db-f37c-4b49-84c6-4f3684d6a295-kube-api-access-d822n\") pod \"maas-api-revoke-sub-bfc8c1d0-rmfkv\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:48.810639 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:48.810605    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:49.200308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:49.200229    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 04:01:49.200308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:49.200286    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 04:01:49.208193 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:49.208170    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"]
Nov 01 04:01:49.210031 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:01:49.210004    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db616db_f37c_4b49_84c6_4f3684d6a295.slice/crio-96077016a73b21cf27202792b7b4c20ce6e91e958aa05ff0c5032ba523893e71 WatchSource:0}: Error finding container 96077016a73b21cf27202792b7b4c20ce6e91e958aa05ff0c5032ba523893e71: Status 404 returned error can't find the container with id 96077016a73b21cf27202792b7b4c20ce6e91e958aa05ff0c5032ba523893e71
Nov 01 04:01:49.494678 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:49.494601    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv" event={"ID":"6db616db-f37c-4b49-84c6-4f3684d6a295","Type":"ContainerStarted","Data":"96077016a73b21cf27202792b7b4c20ce6e91e958aa05ff0c5032ba523893e71"}
Nov 01 04:01:50.200799 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:50.200766    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 04:01:50.200902 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:50.200828    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 04:01:50.503521 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:50.503482    4882 generic.go:356] "Generic (PLEG): container finished" podID="6db616db-f37c-4b49-84c6-4f3684d6a295" containerID="3014a7ff7d6c04abec1366d50b765defe23eac93387dfc869416aa1ef17dd8a8" exitCode=0
Nov 01 04:01:50.503908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:50.503575    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv" event={"ID":"6db616db-f37c-4b49-84c6-4f3684d6a295","Type":"ContainerDied","Data":"3014a7ff7d6c04abec1366d50b765defe23eac93387dfc869416aa1ef17dd8a8"}
Nov 01 04:01:51.200653 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.200618    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 04:01:51.200906 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.200879    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 04:01:51.721011 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.720979    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:51.841524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.839085    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db616db-f37c-4b49-84c6-4f3684d6a295-kube-api-access-d822n\" (UniqueName: \"kubernetes.io/projected/6db616db-f37c-4b49-84c6-4f3684d6a295-kube-api-access-d822n\") pod \"6db616db-f37c-4b49-84c6-4f3684d6a295\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") "
Nov 01 04:01:51.841524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.839138    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-openshift-ca\") pod \"6db616db-f37c-4b49-84c6-4f3684d6a295\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") "
Nov 01 04:01:51.841524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.839224    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-xks-ca\") pod \"6db616db-f37c-4b49-84c6-4f3684d6a295\" (UID: \"6db616db-f37c-4b49-84c6-4f3684d6a295\") "
Nov 01 04:01:51.841524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.840607    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-openshift-ca" pod "6db616db-f37c-4b49-84c6-4f3684d6a295" (UID: "6db616db-f37c-4b49-84c6-4f3684d6a295"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:01:51.846225 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.842702    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db616db-f37c-4b49-84c6-4f3684d6a295-kube-api-access-d822n" pod "6db616db-f37c-4b49-84c6-4f3684d6a295" (UID: "6db616db-f37c-4b49-84c6-4f3684d6a295"). InnerVolumeSpecName "kube-api-access-d822n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:01:51.848004 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.847981    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-xks-ca" pod "6db616db-f37c-4b49-84c6-4f3684d6a295" (UID: "6db616db-f37c-4b49-84c6-4f3684d6a295"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:01:51.941375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.941333    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d822n\" (UniqueName: \"kubernetes.io/projected/6db616db-f37c-4b49-84c6-4f3684d6a295-kube-api-access-d822n\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:51.941550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.941380    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:51.941550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:51.941398    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db616db-f37c-4b49-84c6-4f3684d6a295-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:01:52.200346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:52.200308    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 04:01:52.200532 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:52.200374    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 04:01:52.519691 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:52.519603    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"
Nov 01 04:01:52.519691 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:52.519637    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv" event={"ID":"6db616db-f37c-4b49-84c6-4f3684d6a295","Type":"ContainerDied","Data":"96077016a73b21cf27202792b7b4c20ce6e91e958aa05ff0c5032ba523893e71"}
Nov 01 04:01:52.519870 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:52.519694    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96077016a73b21cf27202792b7b4c20ce6e91e958aa05ff0c5032ba523893e71"
Nov 01 04:01:53.204484 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:53.204449    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:53.204941 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:53.204905    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:53.205637 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:53.205619    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:01:58.146722 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.146688    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"]
Nov 01 04:01:58.147245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.147230    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6db616db-f37c-4b49-84c6-4f3684d6a295" containerName="revoke-keys"
Nov 01 04:01:58.147291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.147248    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db616db-f37c-4b49-84c6-4f3684d6a295" containerName="revoke-keys"
Nov 01 04:01:58.147369 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.147359    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db616db-f37c-4b49-84c6-4f3684d6a295" containerName="revoke-keys"
Nov 01 04:01:58.151024 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.151002    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:01:58.200291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.200267    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:01:58.208320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.208287    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"]
Nov 01 04:01:58.246538 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.246505    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:01:58.272229 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.272196    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnbxn\" (UniqueName: \"kubernetes.io/projected/dbae6694-2406-4fa7-9794-8fea2fb69fe0-kube-api-access-qnbxn\") pod \"maas-api-revoke-sub-659955d3-svhjq\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:01:58.272399 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.272364    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659955d3-svhjq\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:01:58.272518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.272501    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659955d3-svhjq\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:01:58.373329 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.373295    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659955d3-svhjq\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:01:58.373519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.373386    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659955d3-svhjq\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:01:58.373585 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.373520    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnbxn\" (UniqueName: \"kubernetes.io/projected/dbae6694-2406-4fa7-9794-8fea2fb69fe0-kube-api-access-qnbxn\") pod \"maas-api-revoke-sub-659955d3-svhjq\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:01:58.374124 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.374100    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659955d3-svhjq\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:01:58.375614 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.375591    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659955d3-svhjq\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:01:58.430777 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.430712    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnbxn\" (UniqueName: \"kubernetes.io/projected/dbae6694-2406-4fa7-9794-8fea2fb69fe0-kube-api-access-qnbxn\") pod \"maas-api-revoke-sub-659955d3-svhjq\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:01:58.464240 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.464214    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:01:58.726219 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:01:58.726171    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:01:58.841653 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:58.841624    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"]
Nov 01 04:01:58.842403 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:01:58.842376    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbae6694_2406_4fa7_9794_8fea2fb69fe0.slice/crio-48c04646764ed447ad46d1c2434d02adc33f58e0b36e4505a8aaeb92a04d1cd0 WatchSource:0}: Error finding container 48c04646764ed447ad46d1c2434d02adc33f58e0b36e4505a8aaeb92a04d1cd0: Status 404 returned error can't find the container with id 48c04646764ed447ad46d1c2434d02adc33f58e0b36e4505a8aaeb92a04d1cd0
Nov 01 04:01:59.292884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.292855    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"]
Nov 01 04:01:59.297477 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.297452    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:01:59.351669 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.351542    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"]
Nov 01 04:01:59.387043 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.387008    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhzb\" (UniqueName: \"kubernetes.io/projected/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-kube-api-access-snhzb\") pod \"maas-api-revoke-sub-99f4129e-cqqcm\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:01:59.387180 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.387144    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99f4129e-cqqcm\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:01:59.387312 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.387296    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99f4129e-cqqcm\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:01:59.488224 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.488191    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99f4129e-cqqcm\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:01:59.488401 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.488334    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99f4129e-cqqcm\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:01:59.488506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.488422    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhzb\" (UniqueName: \"kubernetes.io/projected/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-kube-api-access-snhzb\") pod \"maas-api-revoke-sub-99f4129e-cqqcm\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:01:59.488833 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.488814    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99f4129e-cqqcm\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:01:59.490711 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.490692    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99f4129e-cqqcm\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:01:59.545652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.545621    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhzb\" (UniqueName: \"kubernetes.io/projected/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-kube-api-access-snhzb\") pod \"maas-api-revoke-sub-99f4129e-cqqcm\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:01:59.611254 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.611183    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:01:59.637632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.637599    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq" event={"ID":"dbae6694-2406-4fa7-9794-8fea2fb69fe0","Type":"ContainerStarted","Data":"3e64177c10fe575bceef13ebbc4f4d9c367fac557fff7cb29448c9d71a196f69"}
Nov 01 04:01:59.637788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.637658    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq" event={"ID":"dbae6694-2406-4fa7-9794-8fea2fb69fe0","Type":"ContainerStarted","Data":"48c04646764ed447ad46d1c2434d02adc33f58e0b36e4505a8aaeb92a04d1cd0"}
Nov 01 04:01:59.748697 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.748642    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq" podStartSLOduration=1.748627195 podStartE2EDuration="1.748627195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:59.746495424 +0000 UTC m=+1188.308528440" watchObservedRunningTime="2026-11-01 04:01:59.748627195 +0000 UTC m=+1188.310660212"
Nov 01 04:01:59.782721 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.782672    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"]
Nov 01 04:01:59.788124 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.788102    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.838665 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.838639    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"distinct-w3-f73b9f-kserve-self-signed-certs\""
Nov 01 04:01:59.862069 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.857849    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"]
Nov 01 04:01:59.893703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.893664    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9md9\" (UniqueName: \"kubernetes.io/projected/97c01b72-75a0-4153-819c-0f37c6b7d427-kube-api-access-l9md9\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.893893 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.893725    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-home\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.893893 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.893867    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-dshm\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.894038 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.893996    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-model-cache\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.894092 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.894037    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-tmp-dir\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.894190 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.894173    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c01b72-75a0-4153-819c-0f37c6b7d427-tls-certs\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.985128 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.985097    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"]
Nov 01 04:01:59.986702 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:01:59.986675    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4f66b2_482e_4ad7_b0d4_f497d440e3ab.slice/crio-62b26e607ff4da932c4a5d956b4461d87e7b4fe70068c8721e5228441b0fb2f5 WatchSource:0}: Error finding container 62b26e607ff4da932c4a5d956b4461d87e7b4fe70068c8721e5228441b0fb2f5: Status 404 returned error can't find the container with id 62b26e607ff4da932c4a5d956b4461d87e7b4fe70068c8721e5228441b0fb2f5
Nov 01 04:01:59.995471 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.995412    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c01b72-75a0-4153-819c-0f37c6b7d427-tls-certs\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.995567 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.995522    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9md9\" (UniqueName: \"kubernetes.io/projected/97c01b72-75a0-4153-819c-0f37c6b7d427-kube-api-access-l9md9\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.995626 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.995562    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-home\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.995738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.995718    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-dshm\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.995804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.995788    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-model-cache\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.995857 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.995821    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-tmp-dir\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.995954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.995934    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-home\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.996096 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.996077    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-model-cache\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.996242 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.996209    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-tmp-dir\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.997884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.997867    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-dshm\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:01:59.998209 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:01:59.998190    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c01b72-75a0-4153-819c-0f37c6b7d427-tls-certs\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:02:00.052492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:00.052466    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9md9\" (UniqueName: \"kubernetes.io/projected/97c01b72-75a0-4153-819c-0f37c6b7d427-kube-api-access-l9md9\") pod \"distinct-w3-f73b9f-kserve-67cc5475-7lsd9\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:02:00.101286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:00.101254    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:02:00.523115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:00.523084    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"]
Nov 01 04:02:00.524024 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:02:00.523999    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97c01b72_75a0_4153_819c_0f37c6b7d427.slice/crio-3f3a77ce0d184a156e5d4779293f36da9e42e2ecb9ec11afb3ae4213339b1e5d WatchSource:0}: Error finding container 3f3a77ce0d184a156e5d4779293f36da9e42e2ecb9ec11afb3ae4213339b1e5d: Status 404 returned error can't find the container with id 3f3a77ce0d184a156e5d4779293f36da9e42e2ecb9ec11afb3ae4213339b1e5d
Nov 01 04:02:00.646379 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:00.646297    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" event={"ID":"97c01b72-75a0-4153-819c-0f37c6b7d427","Type":"ContainerStarted","Data":"11487d72df8cc4ab17a47e5d9d3cd0a28cbf372e18d9eb92fdd8878a900f21f2"}
Nov 01 04:02:00.646379 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:00.646342    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" event={"ID":"97c01b72-75a0-4153-819c-0f37c6b7d427","Type":"ContainerStarted","Data":"3f3a77ce0d184a156e5d4779293f36da9e42e2ecb9ec11afb3ae4213339b1e5d"}
Nov 01 04:02:00.647920 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:00.647891    4882 generic.go:356] "Generic (PLEG): container finished" podID="dbae6694-2406-4fa7-9794-8fea2fb69fe0" containerID="3e64177c10fe575bceef13ebbc4f4d9c367fac557fff7cb29448c9d71a196f69" exitCode=0
Nov 01 04:02:00.648057 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:00.647968    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq" event={"ID":"dbae6694-2406-4fa7-9794-8fea2fb69fe0","Type":"ContainerDied","Data":"3e64177c10fe575bceef13ebbc4f4d9c367fac557fff7cb29448c9d71a196f69"}
Nov 01 04:02:00.649444 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:00.649408    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm" event={"ID":"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab","Type":"ContainerStarted","Data":"260545963f8eebe16d8fb97c906424f5b45142b2573829097b34ee5249cbbb91"}
Nov 01 04:02:00.649566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:00.649458    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm" event={"ID":"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab","Type":"ContainerStarted","Data":"62b26e607ff4da932c4a5d956b4461d87e7b4fe70068c8721e5228441b0fb2f5"}
Nov 01 04:02:00.755156 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:00.755098    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" podStartSLOduration=1.755081221 podStartE2EDuration="1.755081221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:00.754685348 +0000 UTC m=+1189.316718366" watchObservedRunningTime="2026-11-01 04:02:00.755081221 +0000 UTC m=+1189.317114239"
Nov 01 04:02:00.863600 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:00.863548    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm" podStartSLOduration=1.8635311570000002 podStartE2EDuration="1.863531157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:00.861911425 +0000 UTC m=+1189.423944442" watchObservedRunningTime="2026-11-01 04:02:00.863531157 +0000 UTC m=+1189.425564215"
Nov 01 04:02:01.102272 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:01.102246    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:02:01.657498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:01.657462    4882 generic.go:356] "Generic (PLEG): container finished" podID="ec4f66b2-482e-4ad7-b0d4-f497d440e3ab" containerID="260545963f8eebe16d8fb97c906424f5b45142b2573829097b34ee5249cbbb91" exitCode=0
Nov 01 04:02:01.657952 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:01.657467    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm" event={"ID":"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab","Type":"ContainerDied","Data":"260545963f8eebe16d8fb97c906424f5b45142b2573829097b34ee5249cbbb91"}
Nov 01 04:02:01.805558 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:01.805532    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:02:01.922226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:01.922145    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-openshift-ca\") pod \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") "
Nov 01 04:02:01.922401 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:01.922388    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbae6694-2406-4fa7-9794-8fea2fb69fe0-kube-api-access-qnbxn\" (UniqueName: \"kubernetes.io/projected/dbae6694-2406-4fa7-9794-8fea2fb69fe0-kube-api-access-qnbxn\") pod \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") "
Nov 01 04:02:01.922569 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:01.922549    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-xks-ca\") pod \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\" (UID: \"dbae6694-2406-4fa7-9794-8fea2fb69fe0\") "
Nov 01 04:02:01.922629 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:01.922581    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-openshift-ca" pod "dbae6694-2406-4fa7-9794-8fea2fb69fe0" (UID: "dbae6694-2406-4fa7-9794-8fea2fb69fe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:02:01.923051 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:01.923029    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:01.924742 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:01.924709    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-xks-ca" pod "dbae6694-2406-4fa7-9794-8fea2fb69fe0" (UID: "dbae6694-2406-4fa7-9794-8fea2fb69fe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:02:01.925041 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:01.925020    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbae6694-2406-4fa7-9794-8fea2fb69fe0-kube-api-access-qnbxn" pod "dbae6694-2406-4fa7-9794-8fea2fb69fe0" (UID: "dbae6694-2406-4fa7-9794-8fea2fb69fe0"). InnerVolumeSpecName "kube-api-access-qnbxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:02:02.024700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.024669    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnbxn\" (UniqueName: \"kubernetes.io/projected/dbae6694-2406-4fa7-9794-8fea2fb69fe0-kube-api-access-qnbxn\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:02.024700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.024696    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbae6694-2406-4fa7-9794-8fea2fb69fe0-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:02.665064 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.665032    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"
Nov 01 04:02:02.665064 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.665032    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq" event={"ID":"dbae6694-2406-4fa7-9794-8fea2fb69fe0","Type":"ContainerDied","Data":"48c04646764ed447ad46d1c2434d02adc33f58e0b36e4505a8aaeb92a04d1cd0"}
Nov 01 04:02:02.665508 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.665085    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48c04646764ed447ad46d1c2434d02adc33f58e0b36e4505a8aaeb92a04d1cd0"
Nov 01 04:02:02.820596 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.820570    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:02:02.934609 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.934586    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-xks-ca\") pod \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") "
Nov 01 04:02:02.934758 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.934704    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-kube-api-access-snhzb\" (UniqueName: \"kubernetes.io/projected/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-kube-api-access-snhzb\") pod \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") "
Nov 01 04:02:02.934841 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.934831    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-openshift-ca\") pod \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\" (UID: \"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab\") "
Nov 01 04:02:02.935237 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.935208    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-openshift-ca" pod "ec4f66b2-482e-4ad7-b0d4-f497d440e3ab" (UID: "ec4f66b2-482e-4ad7-b0d4-f497d440e3ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:02:02.935345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.935307    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:02.936613 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.936591    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-xks-ca" pod "ec4f66b2-482e-4ad7-b0d4-f497d440e3ab" (UID: "ec4f66b2-482e-4ad7-b0d4-f497d440e3ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:02:02.936829 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:02.936812    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-kube-api-access-snhzb" pod "ec4f66b2-482e-4ad7-b0d4-f497d440e3ab" (UID: "ec4f66b2-482e-4ad7-b0d4-f497d440e3ab"). InnerVolumeSpecName "kube-api-access-snhzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:02:03.036860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:03.036826    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:03.036860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:03.036856    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhzb\" (UniqueName: \"kubernetes.io/projected/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab-kube-api-access-snhzb\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:03.678633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:03.678589    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm" event={"ID":"ec4f66b2-482e-4ad7-b0d4-f497d440e3ab","Type":"ContainerDied","Data":"62b26e607ff4da932c4a5d956b4461d87e7b4fe70068c8721e5228441b0fb2f5"}
Nov 01 04:02:03.678633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:03.678620    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"
Nov 01 04:02:03.679180 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:03.678687    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62b26e607ff4da932c4a5d956b4461d87e7b4fe70068c8721e5228441b0fb2f5"
Nov 01 04:02:07.739111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.739077    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"]
Nov 01 04:02:07.739642 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.739627    4882 cpu_manager.go:414] "containerMap: removing container" podUID="dbae6694-2406-4fa7-9794-8fea2fb69fe0" containerName="revoke-keys"
Nov 01 04:02:07.739642 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.739642    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbae6694-2406-4fa7-9794-8fea2fb69fe0" containerName="revoke-keys"
Nov 01 04:02:07.739748 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.739674    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ec4f66b2-482e-4ad7-b0d4-f497d440e3ab" containerName="revoke-keys"
Nov 01 04:02:07.739748 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.739681    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4f66b2-482e-4ad7-b0d4-f497d440e3ab" containerName="revoke-keys"
Nov 01 04:02:07.739806 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.739780    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4f66b2-482e-4ad7-b0d4-f497d440e3ab" containerName="revoke-keys"
Nov 01 04:02:07.739806 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.739794    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbae6694-2406-4fa7-9794-8fea2fb69fe0" containerName="revoke-keys"
Nov 01 04:02:07.744583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.744564    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:07.839037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.838999    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-865066\"/\"distinct-w2-865066-kserve-self-signed-certs\""
Nov 01 04:02:07.843019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.842989    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"]
Nov 01 04:02:07.909824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.909770    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c92a7a3f-7ff0-42fc-846a-cc386e951275-tls-certs\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:07.910046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.909924    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-home\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:07.910046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.910001    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-dshm\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:07.910157 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.910073    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-model-cache\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:07.910210 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.910170    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn86v\" (UniqueName: \"kubernetes.io/projected/c92a7a3f-7ff0-42fc-846a-cc386e951275-kube-api-access-pn86v\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:07.910269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:07.910207    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-tmp-dir\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.011935 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.011852    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn86v\" (UniqueName: \"kubernetes.io/projected/c92a7a3f-7ff0-42fc-846a-cc386e951275-kube-api-access-pn86v\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.011935 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.011897    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-tmp-dir\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.012164 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.012034    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c92a7a3f-7ff0-42fc-846a-cc386e951275-tls-certs\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.012164 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.012081    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-home\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.012164 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.012135    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-dshm\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.012325 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.012191    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-model-cache\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.012439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.012402    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-tmp-dir\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.012507 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.012453    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-home\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.012626 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.012606    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-model-cache\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.014310 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.014289    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-dshm\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.014621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.014605    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c92a7a3f-7ff0-42fc-846a-cc386e951275-tls-certs\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.069105 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.069075    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn86v\" (UniqueName: \"kubernetes.io/projected/c92a7a3f-7ff0-42fc-846a-cc386e951275-kube-api-access-pn86v\") pod \"distinct-w2-865066-kserve-864548c4cb-jpmps\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") " pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.356612 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.356527    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:08.737956 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:08.737927    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"]
Nov 01 04:02:08.739322 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:02:08.739294    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc92a7a3f_7ff0_42fc_846a_cc386e951275.slice/crio-b23559165e663cf167f4c69bfa4ab85bd52d4c7782261d05f00b049f7dbed792 WatchSource:0}: Error finding container b23559165e663cf167f4c69bfa4ab85bd52d4c7782261d05f00b049f7dbed792: Status 404 returned error can't find the container with id b23559165e663cf167f4c69bfa4ab85bd52d4c7782261d05f00b049f7dbed792
Nov 01 04:02:09.781709 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:09.780630    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" event={"ID":"c92a7a3f-7ff0-42fc-846a-cc386e951275","Type":"ContainerStarted","Data":"932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166"}
Nov 01 04:02:09.781709 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:09.780692    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" event={"ID":"c92a7a3f-7ff0-42fc-846a-cc386e951275","Type":"ContainerStarted","Data":"b23559165e663cf167f4c69bfa4ab85bd52d4c7782261d05f00b049f7dbed792"}
Nov 01 04:02:09.912246 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:09.912179    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" podStartSLOduration=2.912158337 podStartE2EDuration="2.912158337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:09.892945075 +0000 UTC m=+1198.454978118" watchObservedRunningTime="2026-11-01 04:02:09.912158337 +0000 UTC m=+1198.474191357"
Nov 01 04:02:10.106374 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:10.106294    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:02:10.369032 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:10.368939    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:11.105565 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:11.105513    4882 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:11.319573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:11.319525    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:11.352571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:11.352518    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:11.376021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:11.375916    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:11.376471 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:11.376263    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Nov 01 04:02:11.377493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:11.376301    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52" podUID="91fcd874-e74a-4bf5-a6ca-3842296d22b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:11.387557 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:11.387525    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:11.461973 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:11.461916    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:12.155213 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.154078    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:12.162540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.162506    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:02:12.162773 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.162742    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:12.391507 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.387422    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 04:02:12.391507 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.387834    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 01 04:02:12.391507 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.388618    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 04:02:12.391507 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.388696    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 04:02:12.402463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.398549    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:12.402463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.398917    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 04:02:12.402463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.400625    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:12.420470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.404228    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 04:02:12.471452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.467770    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:12.495037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.495010    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 04:02:12.671599 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.657999    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 04:02:12.722529 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.722501    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 04:02:12.725288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.725126    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 04:02:12.789293 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.789263    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 04:02:12.798912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.798088    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 04:02:12.837635 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.837597    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:02:12.838846 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:12.838827    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:02:16.025869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.025835    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"]
Nov 01 04:02:16.031361 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.031337    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:16.070237 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.070206    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406d372b-t6mkb\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:16.070382 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.070261    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjm9\" (UniqueName: \"kubernetes.io/projected/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-kube-api-access-qsjm9\") pod \"maas-api-revoke-sub-406d372b-t6mkb\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:16.070483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.070415    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406d372b-t6mkb\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:16.083958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.083930    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"]
Nov 01 04:02:16.127464 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.125151    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:02:16.127464 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.125301    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:02:16.171193 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.171158    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406d372b-t6mkb\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:16.171358 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.171295    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406d372b-t6mkb\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:16.171358 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.171330    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjm9\" (UniqueName: \"kubernetes.io/projected/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-kube-api-access-qsjm9\") pod \"maas-api-revoke-sub-406d372b-t6mkb\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:16.171955 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.171934    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406d372b-t6mkb\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:16.173559 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.173536    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406d372b-t6mkb\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:16.227947 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.227914    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjm9\" (UniqueName: \"kubernetes.io/projected/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-kube-api-access-qsjm9\") pod \"maas-api-revoke-sub-406d372b-t6mkb\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:16.345291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.345202    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:16.726619 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.726588    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"]
Nov 01 04:02:16.733471 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:02:16.728147    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9e2a38_7d00_45ab_b84b_bf691400a3e9.slice/crio-9e2c15b76b1cba3afe83898468fdbdfc4c83cad7e20af45156a0d1497eed66ad WatchSource:0}: Error finding container 9e2c15b76b1cba3afe83898468fdbdfc4c83cad7e20af45156a0d1497eed66ad: Status 404 returned error can't find the container with id 9e2c15b76b1cba3afe83898468fdbdfc4c83cad7e20af45156a0d1497eed66ad
Nov 01 04:02:16.733471 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.732287    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"]
Nov 01 04:02:16.738225 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.738201    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:16.779682 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.779659    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b1595d-qwtvz\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:16.779817 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.779796    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fpg\" (UniqueName: \"kubernetes.io/projected/df6ffac2-1fff-4e03-b9eb-1ba04654256a-kube-api-access-s7fpg\") pod \"maas-api-revoke-sub-14b1595d-qwtvz\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:16.779920 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.779907    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b1595d-qwtvz\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:16.832541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.832512    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"]
Nov 01 04:02:16.880921 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.880887    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b1595d-qwtvz\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:16.881093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.881035    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fpg\" (UniqueName: \"kubernetes.io/projected/df6ffac2-1fff-4e03-b9eb-1ba04654256a-kube-api-access-s7fpg\") pod \"maas-api-revoke-sub-14b1595d-qwtvz\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:16.881179 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.881163    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b1595d-qwtvz\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:16.881781 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.881755    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b1595d-qwtvz\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:16.883481 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.883460    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b1595d-qwtvz\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:16.938222 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.938195    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fpg\" (UniqueName: \"kubernetes.io/projected/df6ffac2-1fff-4e03-b9eb-1ba04654256a-kube-api-access-s7fpg\") pod \"maas-api-revoke-sub-14b1595d-qwtvz\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:16.969625 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:16.969592    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb" event={"ID":"9c9e2a38-7d00-45ab-b84b-bf691400a3e9","Type":"ContainerStarted","Data":"9e2c15b76b1cba3afe83898468fdbdfc4c83cad7e20af45156a0d1497eed66ad"}
Nov 01 04:02:17.056411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:17.056382    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:17.428883 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:17.428856    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"]
Nov 01 04:02:17.429189 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:02:17.429167    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6ffac2_1fff_4e03_b9eb_1ba04654256a.slice/crio-540d094968ca3c57b464d8ba47e14fdcbea99fa3d0d95d5c185f56b788ff24d2 WatchSource:0}: Error finding container 540d094968ca3c57b464d8ba47e14fdcbea99fa3d0d95d5c185f56b788ff24d2: Status 404 returned error can't find the container with id 540d094968ca3c57b464d8ba47e14fdcbea99fa3d0d95d5c185f56b788ff24d2
Nov 01 04:02:17.978204 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:17.978169    4882 generic.go:356] "Generic (PLEG): container finished" podID="9c9e2a38-7d00-45ab-b84b-bf691400a3e9" containerID="702c735919305e9b956193074ccbd73a7fe091a3aebf5dea346ee1f6d433539a" exitCode=0
Nov 01 04:02:17.978383 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:17.978254    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb" event={"ID":"9c9e2a38-7d00-45ab-b84b-bf691400a3e9","Type":"ContainerDied","Data":"702c735919305e9b956193074ccbd73a7fe091a3aebf5dea346ee1f6d433539a"}
Nov 01 04:02:17.979809 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:17.979784    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz" event={"ID":"df6ffac2-1fff-4e03-b9eb-1ba04654256a","Type":"ContainerStarted","Data":"9fab6f6f5c394793d705e014ec4721c809716ce6340b202fada28e07b29ac2da"}
Nov 01 04:02:17.979931 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:17.979832    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz" event={"ID":"df6ffac2-1fff-4e03-b9eb-1ba04654256a","Type":"ContainerStarted","Data":"540d094968ca3c57b464d8ba47e14fdcbea99fa3d0d95d5c185f56b788ff24d2"}
Nov 01 04:02:18.194401 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:18.194344    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz" podStartSLOduration=2.194328588 podStartE2EDuration="2.194328588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:18.191946051 +0000 UTC m=+1206.753979068" watchObservedRunningTime="2026-11-01 04:02:18.194328588 +0000 UTC m=+1206.756361614"
Nov 01 04:02:18.356869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:18.356795    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:18.370305 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:18.370278    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:18.988575 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:18.988536    4882 generic.go:356] "Generic (PLEG): container finished" podID="df6ffac2-1fff-4e03-b9eb-1ba04654256a" containerID="9fab6f6f5c394793d705e014ec4721c809716ce6340b202fada28e07b29ac2da" exitCode=0
Nov 01 04:02:18.988781 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:18.988623    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz" event={"ID":"df6ffac2-1fff-4e03-b9eb-1ba04654256a","Type":"ContainerDied","Data":"9fab6f6f5c394793d705e014ec4721c809716ce6340b202fada28e07b29ac2da"}
Nov 01 04:02:19.000326 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.000298    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:02:19.146212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.146176    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:19.210105 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.210068    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-openshift-ca\") pod \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") "
Nov 01 04:02:19.210530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.210302    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-kube-api-access-qsjm9\" (UniqueName: \"kubernetes.io/projected/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-kube-api-access-qsjm9\") pod \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") "
Nov 01 04:02:19.210530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.210368    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-xks-ca\") pod \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\" (UID: \"9c9e2a38-7d00-45ab-b84b-bf691400a3e9\") "
Nov 01 04:02:19.210530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.210487    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-openshift-ca" pod "9c9e2a38-7d00-45ab-b84b-bf691400a3e9" (UID: "9c9e2a38-7d00-45ab-b84b-bf691400a3e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:02:19.210782 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.210764    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:19.212400 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.212376    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-xks-ca" pod "9c9e2a38-7d00-45ab-b84b-bf691400a3e9" (UID: "9c9e2a38-7d00-45ab-b84b-bf691400a3e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:02:19.212514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.212485    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-kube-api-access-qsjm9" pod "9c9e2a38-7d00-45ab-b84b-bf691400a3e9" (UID: "9c9e2a38-7d00-45ab-b84b-bf691400a3e9"). InnerVolumeSpecName "kube-api-access-qsjm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:02:19.312618 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.312523    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsjm9\" (UniqueName: \"kubernetes.io/projected/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-kube-api-access-qsjm9\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:19.312618 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.312597    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9e2a38-7d00-45ab-b84b-bf691400a3e9-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:19.997911 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.997874    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb" event={"ID":"9c9e2a38-7d00-45ab-b84b-bf691400a3e9","Type":"ContainerDied","Data":"9e2c15b76b1cba3afe83898468fdbdfc4c83cad7e20af45156a0d1497eed66ad"}
Nov 01 04:02:19.997911 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.997903    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"
Nov 01 04:02:19.998181 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:19.997927    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e2c15b76b1cba3afe83898468fdbdfc4c83cad7e20af45156a0d1497eed66ad"
Nov 01 04:02:20.116580 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.116544    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:02:20.128668 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.128640    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:02:20.164021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.163997    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:20.222610 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.222578    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-openshift-ca\") pod \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") "
Nov 01 04:02:20.223076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.222683    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6ffac2-1fff-4e03-b9eb-1ba04654256a-kube-api-access-s7fpg\" (UniqueName: \"kubernetes.io/projected/df6ffac2-1fff-4e03-b9eb-1ba04654256a-kube-api-access-s7fpg\") pod \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") "
Nov 01 04:02:20.223076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.222782    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-xks-ca\") pod \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\" (UID: \"df6ffac2-1fff-4e03-b9eb-1ba04654256a\") "
Nov 01 04:02:20.223194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.223077    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-openshift-ca" pod "df6ffac2-1fff-4e03-b9eb-1ba04654256a" (UID: "df6ffac2-1fff-4e03-b9eb-1ba04654256a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:02:20.223369 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.223340    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:20.224868 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.224844    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6ffac2-1fff-4e03-b9eb-1ba04654256a-kube-api-access-s7fpg" pod "df6ffac2-1fff-4e03-b9eb-1ba04654256a" (UID: "df6ffac2-1fff-4e03-b9eb-1ba04654256a"). InnerVolumeSpecName "kube-api-access-s7fpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:02:20.224977 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.224905    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-xks-ca" pod "df6ffac2-1fff-4e03-b9eb-1ba04654256a" (UID: "df6ffac2-1fff-4e03-b9eb-1ba04654256a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:02:20.325451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.325316    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7fpg\" (UniqueName: \"kubernetes.io/projected/df6ffac2-1fff-4e03-b9eb-1ba04654256a-kube-api-access-s7fpg\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:20.325451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:20.325350    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df6ffac2-1fff-4e03-b9eb-1ba04654256a-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:21.006446 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:21.006408    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"
Nov 01 04:02:21.006652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:21.006453    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz" event={"ID":"df6ffac2-1fff-4e03-b9eb-1ba04654256a","Type":"ContainerDied","Data":"540d094968ca3c57b464d8ba47e14fdcbea99fa3d0d95d5c185f56b788ff24d2"}
Nov 01 04:02:21.006652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:21.006509    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="540d094968ca3c57b464d8ba47e14fdcbea99fa3d0d95d5c185f56b788ff24d2"
Nov 01 04:02:26.788018 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.787973    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"]
Nov 01 04:02:26.789378 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.789300    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9c9e2a38-7d00-45ab-b84b-bf691400a3e9" containerName="revoke-keys"
Nov 01 04:02:26.789378 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.789326    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9e2a38-7d00-45ab-b84b-bf691400a3e9" containerName="revoke-keys"
Nov 01 04:02:26.789378 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.789356    4882 cpu_manager.go:414] "containerMap: removing container" podUID="df6ffac2-1fff-4e03-b9eb-1ba04654256a" containerName="revoke-keys"
Nov 01 04:02:26.789629 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.789366    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df6ffac2-1fff-4e03-b9eb-1ba04654256a" containerName="revoke-keys"
Nov 01 04:02:26.789880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.789861    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9e2a38-7d00-45ab-b84b-bf691400a3e9" containerName="revoke-keys"
Nov 01 04:02:26.789880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.789880    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="df6ffac2-1fff-4e03-b9eb-1ba04654256a" containerName="revoke-keys"
Nov 01 04:02:26.794052 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.794035    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:26.842698 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.842667    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"]
Nov 01 04:02:26.883865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.883834    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:02:26.884027 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.883958    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:02:26.913792 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.913761    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a486244-rssgb\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:26.913946 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.913813    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhwxv\" (UniqueName: \"kubernetes.io/projected/57baa698-3b7c-4c44-9670-2863f988a88d-kube-api-access-mhwxv\") pod \"maas-api-revoke-sub-3a486244-rssgb\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:26.913996 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:26.913973    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a486244-rssgb\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:27.015783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.015749    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a486244-rssgb\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:27.015953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.015796    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhwxv\" (UniqueName: \"kubernetes.io/projected/57baa698-3b7c-4c44-9670-2863f988a88d-kube-api-access-mhwxv\") pod \"maas-api-revoke-sub-3a486244-rssgb\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:27.015953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.015865    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a486244-rssgb\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:27.016579 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.016554    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a486244-rssgb\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:27.018129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.018102    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a486244-rssgb\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:27.072933 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.072859    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhwxv\" (UniqueName: \"kubernetes.io/projected/57baa698-3b7c-4c44-9670-2863f988a88d-kube-api-access-mhwxv\") pod \"maas-api-revoke-sub-3a486244-rssgb\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:27.110560 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.110525    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:27.485055 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.485026    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"]
Nov 01 04:02:27.486680 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:02:27.486653    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57baa698_3b7c_4c44_9670_2863f988a88d.slice/crio-99513db02a4b2e164d1b89f2b214a96a3d213330b445d8e03b5e07fc79f40043 WatchSource:0}: Error finding container 99513db02a4b2e164d1b89f2b214a96a3d213330b445d8e03b5e07fc79f40043: Status 404 returned error can't find the container with id 99513db02a4b2e164d1b89f2b214a96a3d213330b445d8e03b5e07fc79f40043
Nov 01 04:02:27.495630 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.495607    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"]
Nov 01 04:02:27.501307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.501288    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:27.554667 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.554621    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"]
Nov 01 04:02:27.623100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.623070    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de59a4cf-tphpx\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:27.623259 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.623112    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g752\" (UniqueName: \"kubernetes.io/projected/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-kube-api-access-5g752\") pod \"maas-api-revoke-sub-de59a4cf-tphpx\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:27.623307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.623279    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de59a4cf-tphpx\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:27.724210 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.724173    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de59a4cf-tphpx\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:27.724374 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.724342    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de59a4cf-tphpx\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:27.724448 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.724401    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g752\" (UniqueName: \"kubernetes.io/projected/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-kube-api-access-5g752\") pod \"maas-api-revoke-sub-de59a4cf-tphpx\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:27.724964 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.724926    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de59a4cf-tphpx\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:27.727032 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.727013    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de59a4cf-tphpx\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:27.782790 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.782767    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g752\" (UniqueName: \"kubernetes.io/projected/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-kube-api-access-5g752\") pod \"maas-api-revoke-sub-de59a4cf-tphpx\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:27.831792 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:27.831756    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:28.054403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:28.054321    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb" event={"ID":"57baa698-3b7c-4c44-9670-2863f988a88d","Type":"ContainerStarted","Data":"ce00eebb4dcab8222d0d360ded8ae1251e5b0ca262a3587a430b6b03e9fccfc7"}
Nov 01 04:02:28.054403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:28.054378    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb" event={"ID":"57baa698-3b7c-4c44-9670-2863f988a88d","Type":"ContainerStarted","Data":"99513db02a4b2e164d1b89f2b214a96a3d213330b445d8e03b5e07fc79f40043"}
Nov 01 04:02:28.163416 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:28.163347    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb" podStartSLOduration=2.163330842 podStartE2EDuration="2.163330842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:28.161948946 +0000 UTC m=+1216.723981963" watchObservedRunningTime="2026-11-01 04:02:28.163330842 +0000 UTC m=+1216.725363862"
Nov 01 04:02:28.200318 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:28.200281    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"]
Nov 01 04:02:28.236654 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:02:28.236624    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0defc6_7634_4c8d_8b8a_0f74cb2a3410.slice/crio-6782773167ef6336f80f0297546a41c3e2b21ecdf63a126d18c140051f1fc667 WatchSource:0}: Error finding container 6782773167ef6336f80f0297546a41c3e2b21ecdf63a126d18c140051f1fc667: Status 404 returned error can't find the container with id 6782773167ef6336f80f0297546a41c3e2b21ecdf63a126d18c140051f1fc667
Nov 01 04:02:28.723721 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:02:28.723682    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:02:29.062737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:29.062646    4882 generic.go:356] "Generic (PLEG): container finished" podID="57baa698-3b7c-4c44-9670-2863f988a88d" containerID="ce00eebb4dcab8222d0d360ded8ae1251e5b0ca262a3587a430b6b03e9fccfc7" exitCode=0
Nov 01 04:02:29.063205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:29.062783    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb" event={"ID":"57baa698-3b7c-4c44-9670-2863f988a88d","Type":"ContainerDied","Data":"ce00eebb4dcab8222d0d360ded8ae1251e5b0ca262a3587a430b6b03e9fccfc7"}
Nov 01 04:02:29.064546 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:29.064523    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx" event={"ID":"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410","Type":"ContainerStarted","Data":"526abc36c3e6ed7538b2dae4a751f575dd8428aad0e63c6d892f05da2ae5aed0"}
Nov 01 04:02:29.064656 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:29.064572    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx" event={"ID":"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410","Type":"ContainerStarted","Data":"6782773167ef6336f80f0297546a41c3e2b21ecdf63a126d18c140051f1fc667"}
Nov 01 04:02:29.281150 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:29.281091    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx" podStartSLOduration=2.281070523 podStartE2EDuration="2.281070523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:29.278576718 +0000 UTC m=+1217.840609737" watchObservedRunningTime="2026-11-01 04:02:29.281070523 +0000 UTC m=+1217.843103541"
Nov 01 04:02:30.087412 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:30.083914    4882 generic.go:356] "Generic (PLEG): container finished" podID="ac0defc6-7634-4c8d-8b8a-0f74cb2a3410" containerID="526abc36c3e6ed7538b2dae4a751f575dd8428aad0e63c6d892f05da2ae5aed0" exitCode=0
Nov 01 04:02:30.087412 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:30.084650    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx" event={"ID":"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410","Type":"ContainerDied","Data":"526abc36c3e6ed7538b2dae4a751f575dd8428aad0e63c6d892f05da2ae5aed0"}
Nov 01 04:02:30.950984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:30.950632    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:31.003796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.003708    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-openshift-ca" pod "57baa698-3b7c-4c44-9670-2863f988a88d" (UID: "57baa698-3b7c-4c44-9670-2863f988a88d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:02:31.004062 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.004041    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-openshift-ca\") pod \"57baa698-3b7c-4c44-9670-2863f988a88d\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") "
Nov 01 04:02:31.006853 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.006806    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-xks-ca\") pod \"57baa698-3b7c-4c44-9670-2863f988a88d\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") "
Nov 01 04:02:31.012059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.012038    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57baa698-3b7c-4c44-9670-2863f988a88d-kube-api-access-mhwxv\" (UniqueName: \"kubernetes.io/projected/57baa698-3b7c-4c44-9670-2863f988a88d-kube-api-access-mhwxv\") pod \"57baa698-3b7c-4c44-9670-2863f988a88d\" (UID: \"57baa698-3b7c-4c44-9670-2863f988a88d\") "
Nov 01 04:02:31.032313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.032272    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:31.088453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.085545    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-xks-ca" pod "57baa698-3b7c-4c44-9670-2863f988a88d" (UID: "57baa698-3b7c-4c44-9670-2863f988a88d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:02:31.120123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.120080    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"]
Nov 01 04:02:31.128742 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.120503    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:31.134475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.128618    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:31.138159 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.134984    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57baa698-3b7c-4c44-9670-2863f988a88d-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:31.272452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.198045    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57baa698-3b7c-4c44-9670-2863f988a88d-kube-api-access-mhwxv" pod "57baa698-3b7c-4c44-9670-2863f988a88d" (UID: "57baa698-3b7c-4c44-9670-2863f988a88d"). InnerVolumeSpecName "kube-api-access-mhwxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:02:31.272452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.215984    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"
Nov 01 04:02:31.299696 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.216025    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb" event={"ID":"57baa698-3b7c-4c44-9670-2863f988a88d","Type":"ContainerDied","Data":"99513db02a4b2e164d1b89f2b214a96a3d213330b445d8e03b5e07fc79f40043"}
Nov 01 04:02:31.299696 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.278484    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99513db02a4b2e164d1b89f2b214a96a3d213330b445d8e03b5e07fc79f40043"
Nov 01 04:02:31.362486 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.348517    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d539f4aa-zdzsg"]
Nov 01 04:02:31.376533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.375610    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhwxv\" (UniqueName: \"kubernetes.io/projected/57baa698-3b7c-4c44-9670-2863f988a88d-kube-api-access-mhwxv\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:31.382480 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.380497    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52" podUID="91fcd874-e74a-4bf5-a6ca-3842296d22b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:31.471452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.467523    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:31.691477 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.690852    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:31.723452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.708821    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954ca3b4-985a-4a30-b2ae-18cfed6e6e56" path="/var/lib/kubelet/pods/954ca3b4-985a-4a30-b2ae-18cfed6e6e56/volumes"
Nov 01 04:02:31.771450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.768249    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:31.771450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.769420    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 04:02:31.899625 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.899522    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:31.913695 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:31.913641    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:32.077982 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.077950    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:32.115635 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.115544    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-openshift-ca\") pod \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") "
Nov 01 04:02:32.115635 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.115618    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-xks-ca\") pod \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") "
Nov 01 04:02:32.116132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.115702    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-kube-api-access-5g752\" (UniqueName: \"kubernetes.io/projected/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-kube-api-access-5g752\") pod \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\" (UID: \"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410\") "
Nov 01 04:02:32.116132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.115941    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-openshift-ca" pod "ac0defc6-7634-4c8d-8b8a-0f74cb2a3410" (UID: "ac0defc6-7634-4c8d-8b8a-0f74cb2a3410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:02:32.116222 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.116205    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:32.118074 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.118049    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-kube-api-access-5g752" pod "ac0defc6-7634-4c8d-8b8a-0f74cb2a3410" (UID: "ac0defc6-7634-4c8d-8b8a-0f74cb2a3410"). InnerVolumeSpecName "kube-api-access-5g752". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:02:32.118223 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.118070    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-xks-ca" pod "ac0defc6-7634-4c8d-8b8a-0f74cb2a3410" (UID: "ac0defc6-7634-4c8d-8b8a-0f74cb2a3410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:02:32.217046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.217015    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:32.217046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.217043    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g752\" (UniqueName: \"kubernetes.io/projected/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410-kube-api-access-5g752\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:32.225080 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.225048    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx" event={"ID":"ac0defc6-7634-4c8d-8b8a-0f74cb2a3410","Type":"ContainerDied","Data":"6782773167ef6336f80f0297546a41c3e2b21ecdf63a126d18c140051f1fc667"}
Nov 01 04:02:32.225080 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.225074    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"
Nov 01 04:02:32.225248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:32.225105    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6782773167ef6336f80f0297546a41c3e2b21ecdf63a126d18c140051f1fc667"
Nov 01 04:02:33.171233 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.171200    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"]
Nov 01 04:02:33.171747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.171732    4882 cpu_manager.go:414] "containerMap: removing container" podUID="57baa698-3b7c-4c44-9670-2863f988a88d" containerName="revoke-keys"
Nov 01 04:02:33.171807 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.171748    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57baa698-3b7c-4c44-9670-2863f988a88d" containerName="revoke-keys"
Nov 01 04:02:33.171807 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.171757    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ac0defc6-7634-4c8d-8b8a-0f74cb2a3410" containerName="revoke-keys"
Nov 01 04:02:33.171807 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.171762    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0defc6-7634-4c8d-8b8a-0f74cb2a3410" containerName="revoke-keys"
Nov 01 04:02:33.171946 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.171876    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="57baa698-3b7c-4c44-9670-2863f988a88d" containerName="revoke-keys"
Nov 01 04:02:33.171946 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.171887    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0defc6-7634-4c8d-8b8a-0f74cb2a3410" containerName="revoke-keys"
Nov 01 04:02:33.175783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.175762    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.217774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.217745    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"]
Nov 01 04:02:33.221919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.221900    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.228202 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.228172    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.228308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.228228    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.228408 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.228391    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.228484 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.228449    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.228543 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.228518    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.228617 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.228593    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.228617 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.228619    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.228794 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.228641    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.228794 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.228770    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9gqj\" (UniqueName: \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-kube-api-access-g9gqj\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.269878 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.269820    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"]
Nov 01 04:02:33.320866 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.320834    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"]
Nov 01 04:02:33.329857 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.329811    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.330015 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.329868    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9gqj\" (UniqueName: \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-kube-api-access-g9gqj\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.330015 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.329942    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9j6p\" (UniqueName: \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-kube-api-access-v9j6p\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.330015 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330007    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.330138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330087    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.330138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330113    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.330285 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330266    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.330345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330309    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.330487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330468    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.330598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330495    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.330598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330513    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c799659-ba33-4ce6-826c-58897edd2134-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.330598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330561    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.330598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330564    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.330819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330660    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.330819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330662    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c799659-ba33-4ce6-826c-58897edd2134-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.330819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330751    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.330819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330781    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.331035 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330818    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.331035 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330830    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.331035 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330846    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.331035 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330856    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.331035 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.330933    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.331521 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.331499    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.333145 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.333059    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.333652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.333630    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.391813 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.391780    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9gqj\" (UniqueName: \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-kube-api-access-g9gqj\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.391940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.391868    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.432090 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432005    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432090 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432061    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432322 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432190    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c799659-ba33-4ce6-826c-58897edd2134-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432322 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432276    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c799659-ba33-4ce6-826c-58897edd2134-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432328    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432380    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432456    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432485    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432518    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9j6p\" (UniqueName: \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-kube-api-access-v9j6p\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432555    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432730    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432844 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432809    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.432988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.432969    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.433061 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.433033    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c799659-ba33-4ce6-826c-58897edd2134-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.434847 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.434822    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c799659-ba33-4ce6-826c-58897edd2134-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.434963 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.434868    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.489004 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.488967    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:33.489232 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.489214    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.489290 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.489271    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9j6p\" (UniqueName: \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-kube-api-access-v9j6p\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.545781 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.545709    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:33.915164 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.915122    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"]
Nov 01 04:02:33.950379 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:02:33.950348    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf715c6b5_edcb_46d9_8ed0_12c09b5164e7.slice/crio-50730afc25f74f74d0f42208d95e8a18dfb3a9c2d9fa1167bfe3813c6514b965 WatchSource:0}: Error finding container 50730afc25f74f74d0f42208d95e8a18dfb3a9c2d9fa1167bfe3813c6514b965: Status 404 returned error can't find the container with id 50730afc25f74f74d0f42208d95e8a18dfb3a9c2d9fa1167bfe3813c6514b965
Nov 01 04:02:33.952759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.952727    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:33.952834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.952795    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:33.952871 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.952840    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:33.965377 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.965348    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"]
Nov 01 04:02:33.968804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.968777    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:33.968869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.968837    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:33.968911 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:33.968889    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:34.245274 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:34.245231    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" event={"ID":"6c799659-ba33-4ce6-826c-58897edd2134","Type":"ContainerStarted","Data":"0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27"}
Nov 01 04:02:34.245786 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:34.245291    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" event={"ID":"6c799659-ba33-4ce6-826c-58897edd2134","Type":"ContainerStarted","Data":"595eb225af2fe9979273aa0699eed19d7b049c8d5140e91f358b75ef46ebfa10"}
Nov 01 04:02:34.247412 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:34.247387    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" event={"ID":"f715c6b5-edcb-46d9-8ed0-12c09b5164e7","Type":"ContainerStarted","Data":"a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9"}
Nov 01 04:02:34.247569 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:34.247457    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" event={"ID":"f715c6b5-edcb-46d9-8ed0-12c09b5164e7","Type":"ContainerStarted","Data":"50730afc25f74f74d0f42208d95e8a18dfb3a9c2d9fa1167bfe3813c6514b965"}
Nov 01 04:02:34.358449 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:34.358370    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" podStartSLOduration=1.358352461 podStartE2EDuration="1.358352461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:34.357093576 +0000 UTC m=+1222.919126593" watchObservedRunningTime="2026-11-01 04:02:34.358352461 +0000 UTC m=+1222.920385478"
Nov 01 04:02:34.474827 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:34.474752    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" podStartSLOduration=1.474731391 podStartE2EDuration="1.474731391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:34.469905013 +0000 UTC m=+1223.031938034" watchObservedRunningTime="2026-11-01 04:02:34.474731391 +0000 UTC m=+1223.036764409"
Nov 01 04:02:34.499508 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:34.492568    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:34.545915 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:34.545877    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:35.508451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:35.507554    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 01 04:02:35.508451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:35.507622    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 01 04:02:35.547989 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:35.547947    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 01 04:02:35.548158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:35.548009    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" podUID="6c799659-ba33-4ce6-826c-58897edd2134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 01 04:02:36.498897 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:36.498834    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 01 04:02:36.500805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:36.500766    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 01 04:02:36.546509 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:36.546479    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 01 04:02:36.546959 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:36.546934    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" podUID="6c799659-ba33-4ce6-826c-58897edd2134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 01 04:02:37.493451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:37.493041    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 01 04:02:37.493451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:37.493117    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 01 04:02:37.546530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:37.546497    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 01 04:02:37.547005 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:37.546979    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" podUID="6c799659-ba33-4ce6-826c-58897edd2134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 01 04:02:38.491853 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:38.491815    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 01 04:02:38.492131 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:38.492104    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 01 04:02:38.546274 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:38.546239    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 01 04:02:38.546417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:38.546296    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" podUID="6c799659-ba33-4ce6-826c-58897edd2134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 01 04:02:39.492452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:39.489767    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 01 04:02:39.492452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:39.489853    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 01 04:02:39.546047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:39.546014    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 01 04:02:39.546273 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:39.546254    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" podUID="6c799659-ba33-4ce6-826c-58897edd2134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 01 04:02:40.340041 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.340003    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"]
Nov 01 04:02:40.345612 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.345588    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:40.399368 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.399335    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"]
Nov 01 04:02:40.439539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.439508    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:02:40.439738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.439559    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:02:40.490407 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.490354    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 01 04:02:40.490621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.490455    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 01 04:02:40.507394 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.507355    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36db9bea-fbwlb\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:40.507915 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.507487    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n64l\" (UniqueName: \"kubernetes.io/projected/a4536851-8c8a-4dab-9766-9fa9d1ad157a-kube-api-access-2n64l\") pod \"maas-api-revoke-sub-36db9bea-fbwlb\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:40.507915 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.507531    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36db9bea-fbwlb\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:40.546879 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.546843    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 01 04:02:40.547042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.546922    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" podUID="6c799659-ba33-4ce6-826c-58897edd2134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 01 04:02:40.609023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.608922    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36db9bea-fbwlb\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:40.609023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.608983    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n64l\" (UniqueName: \"kubernetes.io/projected/a4536851-8c8a-4dab-9766-9fa9d1ad157a-kube-api-access-2n64l\") pod \"maas-api-revoke-sub-36db9bea-fbwlb\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:40.609259 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.609105    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36db9bea-fbwlb\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:40.609910 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.609881    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36db9bea-fbwlb\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:40.611660 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.611636    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36db9bea-fbwlb\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:40.666261 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.666229    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n64l\" (UniqueName: \"kubernetes.io/projected/a4536851-8c8a-4dab-9766-9fa9d1ad157a-kube-api-access-2n64l\") pod \"maas-api-revoke-sub-36db9bea-fbwlb\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:40.960870 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:40.960826    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:41.404513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:41.404486    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"]
Nov 01 04:02:41.406801 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:02:41.406765    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4536851_8c8a_4dab_9766_9fa9d1ad157a.slice/crio-39b0dfd6d7ad2bf428c319ff5772fe28214d30140ee81404ef3028b717b5b669 WatchSource:0}: Error finding container 39b0dfd6d7ad2bf428c319ff5772fe28214d30140ee81404ef3028b717b5b669: Status 404 returned error can't find the container with id 39b0dfd6d7ad2bf428c319ff5772fe28214d30140ee81404ef3028b717b5b669
Nov 01 04:02:41.494093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:41.494058    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:41.494501 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:41.494467    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:41.495171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:41.495148    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:02:41.550532 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:41.550501    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:41.550967 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:41.550774    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:41.551366 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:41.551351    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:02:42.358421 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:42.358380    4882 generic.go:356] "Generic (PLEG): container finished" podID="a4536851-8c8a-4dab-9766-9fa9d1ad157a" containerID="ae2ed16beaf71c016634059288c0ff991e05cf35d0913a62dcb0f439d8fb427f" exitCode=0
Nov 01 04:02:42.358723 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:42.358474    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb" event={"ID":"a4536851-8c8a-4dab-9766-9fa9d1ad157a","Type":"ContainerDied","Data":"ae2ed16beaf71c016634059288c0ff991e05cf35d0913a62dcb0f439d8fb427f"}
Nov 01 04:02:42.358723 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:42.358523    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb" event={"ID":"a4536851-8c8a-4dab-9766-9fa9d1ad157a","Type":"ContainerStarted","Data":"39b0dfd6d7ad2bf428c319ff5772fe28214d30140ee81404ef3028b717b5b669"}
Nov 01 04:02:43.516077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:43.516050    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:43.655635 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:43.655547    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-xks-ca\") pod \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") "
Nov 01 04:02:43.655635 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:43.655610    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4536851-8c8a-4dab-9766-9fa9d1ad157a-kube-api-access-2n64l\" (UniqueName: \"kubernetes.io/projected/a4536851-8c8a-4dab-9766-9fa9d1ad157a-kube-api-access-2n64l\") pod \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") "
Nov 01 04:02:43.655876 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:43.655657    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-openshift-ca\") pod \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\" (UID: \"a4536851-8c8a-4dab-9766-9fa9d1ad157a\") "
Nov 01 04:02:43.656077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:43.656052    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-openshift-ca" pod "a4536851-8c8a-4dab-9766-9fa9d1ad157a" (UID: "a4536851-8c8a-4dab-9766-9fa9d1ad157a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:02:43.657913 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:43.657878    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4536851-8c8a-4dab-9766-9fa9d1ad157a-kube-api-access-2n64l" pod "a4536851-8c8a-4dab-9766-9fa9d1ad157a" (UID: "a4536851-8c8a-4dab-9766-9fa9d1ad157a"). InnerVolumeSpecName "kube-api-access-2n64l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:02:43.657913 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:43.657903    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-xks-ca" pod "a4536851-8c8a-4dab-9766-9fa9d1ad157a" (UID: "a4536851-8c8a-4dab-9766-9fa9d1ad157a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:02:43.757811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:43.757768    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n64l\" (UniqueName: \"kubernetes.io/projected/a4536851-8c8a-4dab-9766-9fa9d1ad157a-kube-api-access-2n64l\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:43.757811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:43.757805    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:43.757811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:43.757818    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4536851-8c8a-4dab-9766-9fa9d1ad157a-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:44.375456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.375402    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"
Nov 01 04:02:44.375456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.375446    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb" event={"ID":"a4536851-8c8a-4dab-9766-9fa9d1ad157a","Type":"ContainerDied","Data":"39b0dfd6d7ad2bf428c319ff5772fe28214d30140ee81404ef3028b717b5b669"}
Nov 01 04:02:44.375668 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.375517    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39b0dfd6d7ad2bf428c319ff5772fe28214d30140ee81404ef3028b717b5b669"
Nov 01 04:02:44.898471 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.898422    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"]
Nov 01 04:02:44.899103 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.899069    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a4536851-8c8a-4dab-9766-9fa9d1ad157a" containerName="revoke-keys"
Nov 01 04:02:44.899103 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.899092    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4536851-8c8a-4dab-9766-9fa9d1ad157a" containerName="revoke-keys"
Nov 01 04:02:44.899294 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.899232    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4536851-8c8a-4dab-9766-9fa9d1ad157a" containerName="revoke-keys"
Nov 01 04:02:44.903352 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.903331    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:44.954360 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.954320    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:02:44.954552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.954523    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:02:44.959734 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.959709    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"]
Nov 01 04:02:44.972959 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.972919    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-931cb99d-wbkx4\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:44.973121 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.973072    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9km\" (UniqueName: \"kubernetes.io/projected/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-kube-api-access-xq9km\") pod \"maas-api-revoke-sub-931cb99d-wbkx4\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:44.973213 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:44.973184    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-931cb99d-wbkx4\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:45.074414 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:45.074371    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-931cb99d-wbkx4\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:45.074645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:45.074501    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9km\" (UniqueName: \"kubernetes.io/projected/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-kube-api-access-xq9km\") pod \"maas-api-revoke-sub-931cb99d-wbkx4\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:45.074645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:45.074617    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-931cb99d-wbkx4\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:45.075092 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:45.075067    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-931cb99d-wbkx4\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:45.076882 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:45.076863    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-931cb99d-wbkx4\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:45.131439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:45.131398    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9km\" (UniqueName: \"kubernetes.io/projected/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-kube-api-access-xq9km\") pod \"maas-api-revoke-sub-931cb99d-wbkx4\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:45.219917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:45.219887    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:45.598860 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:02:45.598817    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc8e0d7_3eb5_4a9c_a098_c283ca327e22.slice/crio-0c422a613a21d568a7451c68fb9f99a5e83a6bb06d054ba7a9c721cb39cd0e80 WatchSource:0}: Error finding container 0c422a613a21d568a7451c68fb9f99a5e83a6bb06d054ba7a9c721cb39cd0e80: Status 404 returned error can't find the container with id 0c422a613a21d568a7451c68fb9f99a5e83a6bb06d054ba7a9c721cb39cd0e80
Nov 01 04:02:45.603741 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:45.603715    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"]
Nov 01 04:02:46.501463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:46.501190    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4" event={"ID":"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22","Type":"ContainerStarted","Data":"3f0cef7e0b76be0c446d68b1dd97f2aa1baa22b26d51dcad877a539ef5262698"}
Nov 01 04:02:46.501463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:46.501259    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4" event={"ID":"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22","Type":"ContainerStarted","Data":"0c422a613a21d568a7451c68fb9f99a5e83a6bb06d054ba7a9c721cb39cd0e80"}
Nov 01 04:02:47.111805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:47.097585    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4" podStartSLOduration=3.073641604 podStartE2EDuration="3.073641604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:46.632292825 +0000 UTC m=+1235.194325846" watchObservedRunningTime="2026-11-01 04:02:47.073641604 +0000 UTC m=+1235.635674620"
Nov 01 04:02:47.111805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:47.098711    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"]
Nov 01 04:02:47.160034 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:47.159994    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5437d00e-2dlbc"]
Nov 01 04:02:47.635247 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:02:47.635205    4882 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc8e0d7_3eb5_4a9c_a098_c283ca327e22.slice/crio-conmon-3f0cef7e0b76be0c446d68b1dd97f2aa1baa22b26d51dcad877a539ef5262698.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 04:02:47.668710 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:47.641402    4882 generic.go:356] "Generic (PLEG): container finished" podID="dcc8e0d7-3eb5-4a9c-a098-c283ca327e22" containerID="3f0cef7e0b76be0c446d68b1dd97f2aa1baa22b26d51dcad877a539ef5262698" exitCode=0
Nov 01 04:02:47.696050 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:47.695407    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae" path="/var/lib/kubelet/pods/8d6be10b-6de2-4c8c-a0a6-684c9eb5a3ae/volumes"
Nov 01 04:02:47.696768 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:47.696740    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4" event={"ID":"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22","Type":"ContainerDied","Data":"3f0cef7e0b76be0c446d68b1dd97f2aa1baa22b26d51dcad877a539ef5262698"}
Nov 01 04:02:47.741679 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:47.740383    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 01 04:02:47.820564 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:47.820519    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:47.926624 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:47.921521    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:48.002453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:47.999623    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:48.125006 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:48.124965    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Nov 01 04:02:48.384588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:48.384510    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:02:48.845971 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:48.845947    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:48.919962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:48.919934    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-kube-api-access-xq9km\" (UniqueName: \"kubernetes.io/projected/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-kube-api-access-xq9km\") pod \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") "
Nov 01 04:02:48.920145 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:48.920125    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-xks-ca\") pod \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") "
Nov 01 04:02:48.920236 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:48.920220    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-openshift-ca\") pod \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\" (UID: \"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22\") "
Nov 01 04:02:48.920587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:48.920560    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-openshift-ca" pod "dcc8e0d7-3eb5-4a9c-a098-c283ca327e22" (UID: "dcc8e0d7-3eb5-4a9c-a098-c283ca327e22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:02:48.920795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:48.920772    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:48.922079 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:48.922052    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-kube-api-access-xq9km" pod "dcc8e0d7-3eb5-4a9c-a098-c283ca327e22" (UID: "dcc8e0d7-3eb5-4a9c-a098-c283ca327e22"). InnerVolumeSpecName "kube-api-access-xq9km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:02:48.922235 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:48.922219    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-xks-ca" pod "dcc8e0d7-3eb5-4a9c-a098-c283ca327e22" (UID: "dcc8e0d7-3eb5-4a9c-a098-c283ca327e22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:02:49.021762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:49.021731    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9km\" (UniqueName: \"kubernetes.io/projected/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-kube-api-access-xq9km\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:49.021762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:49.021759    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:49.689186 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:49.689155    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"
Nov 01 04:02:49.689339 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:49.689185    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4" event={"ID":"dcc8e0d7-3eb5-4a9c-a098-c283ca327e22","Type":"ContainerDied","Data":"0c422a613a21d568a7451c68fb9f99a5e83a6bb06d054ba7a9c721cb39cd0e80"}
Nov 01 04:02:49.689339 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:49.689236    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c422a613a21d568a7451c68fb9f99a5e83a6bb06d054ba7a9c721cb39cd0e80"
Nov 01 04:02:54.248106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.248069    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"]
Nov 01 04:02:54.248826 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.248803    4882 cpu_manager.go:414] "containerMap: removing container" podUID="dcc8e0d7-3eb5-4a9c-a098-c283ca327e22" containerName="revoke-keys"
Nov 01 04:02:54.248892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.248829    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc8e0d7-3eb5-4a9c-a098-c283ca327e22" containerName="revoke-keys"
Nov 01 04:02:54.248971 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.248961    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc8e0d7-3eb5-4a9c-a098-c283ca327e22" containerName="revoke-keys"
Nov 01 04:02:54.252863 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.252838    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:54.288411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.288378    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2087ee9d-5v2jh\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:54.288623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.288600    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2087ee9d-5v2jh\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:54.288708 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.288633    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9m4f\" (UniqueName: \"kubernetes.io/projected/98fd2689-9333-40a0-9045-2c5e7a191a98-kube-api-access-v9m4f\") pod \"maas-api-revoke-sub-2087ee9d-5v2jh\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:54.307628 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.307595    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"]
Nov 01 04:02:54.347713 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.347686    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:02:54.347894 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.347766    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:02:54.389550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.389514    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2087ee9d-5v2jh\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:54.389550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.389552    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9m4f\" (UniqueName: \"kubernetes.io/projected/98fd2689-9333-40a0-9045-2c5e7a191a98-kube-api-access-v9m4f\") pod \"maas-api-revoke-sub-2087ee9d-5v2jh\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:54.389819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.389636    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2087ee9d-5v2jh\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:54.390336 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.390317    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2087ee9d-5v2jh\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:54.392048 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.392022    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2087ee9d-5v2jh\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:54.451015 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.450307    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9m4f\" (UniqueName: \"kubernetes.io/projected/98fd2689-9333-40a0-9045-2c5e7a191a98-kube-api-access-v9m4f\") pod \"maas-api-revoke-sub-2087ee9d-5v2jh\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:54.566931 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.566856    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:54.947438 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:54.947402    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"]
Nov 01 04:02:54.948049 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:02:54.948019    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fd2689_9333_40a0_9045_2c5e7a191a98.slice/crio-0a09703ebb201cf7434b0bfe50781e62495d0f254beee2c5443cf8f289bb722c WatchSource:0}: Error finding container 0a09703ebb201cf7434b0bfe50781e62495d0f254beee2c5443cf8f289bb722c: Status 404 returned error can't find the container with id 0a09703ebb201cf7434b0bfe50781e62495d0f254beee2c5443cf8f289bb722c
Nov 01 04:02:55.708955 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.708919    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"]
Nov 01 04:02:55.713234 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.713212    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:55.773756 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.773722    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh" event={"ID":"98fd2689-9333-40a0-9045-2c5e7a191a98","Type":"ContainerStarted","Data":"3ce6856118ea537c1fc3c69c72728e1bed57ceac015e07263379d213a4984396"}
Nov 01 04:02:55.773943 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.773818    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh" event={"ID":"98fd2689-9333-40a0-9045-2c5e7a191a98","Type":"ContainerStarted","Data":"0a09703ebb201cf7434b0bfe50781e62495d0f254beee2c5443cf8f289bb722c"}
Nov 01 04:02:55.778152 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.778124    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"]
Nov 01 04:02:55.808668 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.808629    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e79d56ee-zglcz\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:55.808859 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.808676    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5fx4\" (UniqueName: \"kubernetes.io/projected/1df4f391-88d7-410c-be67-aaa521a42547-kube-api-access-f5fx4\") pod \"maas-api-revoke-sub-e79d56ee-zglcz\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:55.808960 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.808935    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e79d56ee-zglcz\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:55.910443 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.910384    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e79d56ee-zglcz\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:55.910443 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.910446    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5fx4\" (UniqueName: \"kubernetes.io/projected/1df4f391-88d7-410c-be67-aaa521a42547-kube-api-access-f5fx4\") pod \"maas-api-revoke-sub-e79d56ee-zglcz\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:55.910652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.910531    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e79d56ee-zglcz\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:55.911235 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.911214    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e79d56ee-zglcz\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:55.912798 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.912780    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e79d56ee-zglcz\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:55.939888 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.939834    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh" podStartSLOduration=1.939822859 podStartE2EDuration="1.939822859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:55.936512839 +0000 UTC m=+1244.498545856" watchObservedRunningTime="2026-11-01 04:02:55.939822859 +0000 UTC m=+1244.501855876"
Nov 01 04:02:55.967340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:55.967271    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5fx4\" (UniqueName: \"kubernetes.io/projected/1df4f391-88d7-410c-be67-aaa521a42547-kube-api-access-f5fx4\") pod \"maas-api-revoke-sub-e79d56ee-zglcz\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:56.029912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:56.029875    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:56.403115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:56.403040    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"]
Nov 01 04:02:56.407882 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:02:56.407845    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df4f391_88d7_410c_be67_aaa521a42547.slice/crio-ef61c11199da9d5263d9c4701b17f9884ef503772bf0bf02f0559717b223c46f WatchSource:0}: Error finding container ef61c11199da9d5263d9c4701b17f9884ef503772bf0bf02f0559717b223c46f: Status 404 returned error can't find the container with id ef61c11199da9d5263d9c4701b17f9884ef503772bf0bf02f0559717b223c46f
Nov 01 04:02:56.782582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:56.782531    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz" event={"ID":"1df4f391-88d7-410c-be67-aaa521a42547","Type":"ContainerStarted","Data":"48463e31610292fac624333bfe12f413010faa963957ed80ca4917f4f8fc144a"}
Nov 01 04:02:56.783023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:56.782613    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz" event={"ID":"1df4f391-88d7-410c-be67-aaa521a42547","Type":"ContainerStarted","Data":"ef61c11199da9d5263d9c4701b17f9884ef503772bf0bf02f0559717b223c46f"}
Nov 01 04:02:56.784261 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:56.784233    4882 generic.go:356] "Generic (PLEG): container finished" podID="98fd2689-9333-40a0-9045-2c5e7a191a98" containerID="3ce6856118ea537c1fc3c69c72728e1bed57ceac015e07263379d213a4984396" exitCode=0
Nov 01 04:02:56.784361 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:56.784281    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh" event={"ID":"98fd2689-9333-40a0-9045-2c5e7a191a98","Type":"ContainerDied","Data":"3ce6856118ea537c1fc3c69c72728e1bed57ceac015e07263379d213a4984396"}
Nov 01 04:02:56.892922 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:56.892870    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz" podStartSLOduration=1.8928541060000001 podStartE2EDuration="1.892854106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:56.88968284 +0000 UTC m=+1245.451715856" watchObservedRunningTime="2026-11-01 04:02:56.892854106 +0000 UTC m=+1245.454887159"
Nov 01 04:02:57.794552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:57.794509    4882 generic.go:356] "Generic (PLEG): container finished" podID="1df4f391-88d7-410c-be67-aaa521a42547" containerID="48463e31610292fac624333bfe12f413010faa963957ed80ca4917f4f8fc144a" exitCode=0
Nov 01 04:02:57.794965 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:57.794587    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz" event={"ID":"1df4f391-88d7-410c-be67-aaa521a42547","Type":"ContainerDied","Data":"48463e31610292fac624333bfe12f413010faa963957ed80ca4917f4f8fc144a"}
Nov 01 04:02:57.959481 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:57.959455    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:58.037560 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.037528    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-xks-ca\") pod \"98fd2689-9333-40a0-9045-2c5e7a191a98\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") "
Nov 01 04:02:58.037721 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.037611    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98fd2689-9333-40a0-9045-2c5e7a191a98-kube-api-access-v9m4f\" (UniqueName: \"kubernetes.io/projected/98fd2689-9333-40a0-9045-2c5e7a191a98-kube-api-access-v9m4f\") pod \"98fd2689-9333-40a0-9045-2c5e7a191a98\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") "
Nov 01 04:02:58.037721 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.037710    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-openshift-ca\") pod \"98fd2689-9333-40a0-9045-2c5e7a191a98\" (UID: \"98fd2689-9333-40a0-9045-2c5e7a191a98\") "
Nov 01 04:02:58.038108 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.038078    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-openshift-ca" pod "98fd2689-9333-40a0-9045-2c5e7a191a98" (UID: "98fd2689-9333-40a0-9045-2c5e7a191a98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:02:58.038312 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.038295    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:58.039614 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.039591    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-xks-ca" pod "98fd2689-9333-40a0-9045-2c5e7a191a98" (UID: "98fd2689-9333-40a0-9045-2c5e7a191a98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:02:58.039693 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.039642    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fd2689-9333-40a0-9045-2c5e7a191a98-kube-api-access-v9m4f" pod "98fd2689-9333-40a0-9045-2c5e7a191a98" (UID: "98fd2689-9333-40a0-9045-2c5e7a191a98"). InnerVolumeSpecName "kube-api-access-v9m4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:02:58.053759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.053721    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"]
Nov 01 04:02:58.057896 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.057871    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"]
Nov 01 04:02:58.061391 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.061372    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11615f6e-gt2mm"]
Nov 01 04:02:58.064750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.064726    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0c667b-vjmlp"]
Nov 01 04:02:58.139197 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.139168    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fd2689-9333-40a0-9045-2c5e7a191a98-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:58.139197 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.139194    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9m4f\" (UniqueName: \"kubernetes.io/projected/98fd2689-9333-40a0-9045-2c5e7a191a98-kube-api-access-v9m4f\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:58.725387 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:02:58.725350    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:02:58.809525 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.809494    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"
Nov 01 04:02:58.812189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.809499    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh" event={"ID":"98fd2689-9333-40a0-9045-2c5e7a191a98","Type":"ContainerDied","Data":"0a09703ebb201cf7434b0bfe50781e62495d0f254beee2c5443cf8f289bb722c"}
Nov 01 04:02:58.812306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.812291    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a09703ebb201cf7434b0bfe50781e62495d0f254beee2c5443cf8f289bb722c"
Nov 01 04:02:58.966395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:58.966365    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:59.051146 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.051064    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-xks-ca\") pod \"1df4f391-88d7-410c-be67-aaa521a42547\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") "
Nov 01 04:02:59.051293 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.051248    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df4f391-88d7-410c-be67-aaa521a42547-kube-api-access-f5fx4\" (UniqueName: \"kubernetes.io/projected/1df4f391-88d7-410c-be67-aaa521a42547-kube-api-access-f5fx4\") pod \"1df4f391-88d7-410c-be67-aaa521a42547\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") "
Nov 01 04:02:59.051293 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.051266    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-openshift-ca\") pod \"1df4f391-88d7-410c-be67-aaa521a42547\" (UID: \"1df4f391-88d7-410c-be67-aaa521a42547\") "
Nov 01 04:02:59.051768 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.051736    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-openshift-ca" pod "1df4f391-88d7-410c-be67-aaa521a42547" (UID: "1df4f391-88d7-410c-be67-aaa521a42547"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:02:59.053202 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.053165    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-xks-ca" pod "1df4f391-88d7-410c-be67-aaa521a42547" (UID: "1df4f391-88d7-410c-be67-aaa521a42547"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:02:59.053302 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.053275    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df4f391-88d7-410c-be67-aaa521a42547-kube-api-access-f5fx4" pod "1df4f391-88d7-410c-be67-aaa521a42547" (UID: "1df4f391-88d7-410c-be67-aaa521a42547"). InnerVolumeSpecName "kube-api-access-f5fx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:02:59.152998 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.152961    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5fx4\" (UniqueName: \"kubernetes.io/projected/1df4f391-88d7-410c-be67-aaa521a42547-kube-api-access-f5fx4\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:59.152998 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.152995    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:59.152998 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.153006    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df4f391-88d7-410c-be67-aaa521a42547-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:02:59.601829 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.601782    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48fb3d3c-7149-4c11-af83-24eb4efc05d1" path="/var/lib/kubelet/pods/48fb3d3c-7149-4c11-af83-24eb4efc05d1/volumes"
Nov 01 04:02:59.602416 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.602390    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b261d8c5-5fc0-4efa-b929-e8ba71f46e7c" path="/var/lib/kubelet/pods/b261d8c5-5fc0-4efa-b929-e8ba71f46e7c/volumes"
Nov 01 04:02:59.817787 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.817761    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"
Nov 01 04:02:59.818167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.817793    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz" event={"ID":"1df4f391-88d7-410c-be67-aaa521a42547","Type":"ContainerDied","Data":"ef61c11199da9d5263d9c4701b17f9884ef503772bf0bf02f0559717b223c46f"}
Nov 01 04:02:59.818167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:02:59.817847    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef61c11199da9d5263d9c4701b17f9884ef503772bf0bf02f0559717b223c46f"
Nov 01 04:03:02.047115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:02.047081    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"]
Nov 01 04:03:02.093203 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:02.093167    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c022b9e0-qrw6b"]
Nov 01 04:03:03.601037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:03.600955    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ab814f-40d0-4ef3-bf2d-0c8029341145" path="/var/lib/kubelet/pods/22ab814f-40d0-4ef3-bf2d-0c8029341145/volumes"
Nov 01 04:03:05.007929 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.007878    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"]
Nov 01 04:03:05.008551 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.008532    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1df4f391-88d7-410c-be67-aaa521a42547" containerName="revoke-keys"
Nov 01 04:03:05.008551 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.008550    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df4f391-88d7-410c-be67-aaa521a42547" containerName="revoke-keys"
Nov 01 04:03:05.008695 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.008564    4882 cpu_manager.go:414] "containerMap: removing container" podUID="98fd2689-9333-40a0-9045-2c5e7a191a98" containerName="revoke-keys"
Nov 01 04:03:05.008695 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.008569    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98fd2689-9333-40a0-9045-2c5e7a191a98" containerName="revoke-keys"
Nov 01 04:03:05.008695 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.008680    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df4f391-88d7-410c-be67-aaa521a42547" containerName="revoke-keys"
Nov 01 04:03:05.008695 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.008690    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="98fd2689-9333-40a0-9045-2c5e7a191a98" containerName="revoke-keys"
Nov 01 04:03:05.012499 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.012475    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:05.067568 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.067531    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"]
Nov 01 04:03:05.106792 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.106756    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:03:05.106973 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.106857    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:03:05.130811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.130772    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb96eb5d-6zm9c\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:05.131085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.130970    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz87q\" (UniqueName: \"kubernetes.io/projected/be5fb81f-7fe6-49b6-aa63-39d0f161104f-kube-api-access-pz87q\") pod \"maas-api-revoke-sub-eb96eb5d-6zm9c\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:05.131085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.131021    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb96eb5d-6zm9c\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:05.254450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.248841    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz87q\" (UniqueName: \"kubernetes.io/projected/be5fb81f-7fe6-49b6-aa63-39d0f161104f-kube-api-access-pz87q\") pod \"maas-api-revoke-sub-eb96eb5d-6zm9c\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:05.254450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.248901    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb96eb5d-6zm9c\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:05.254450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.249063    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb96eb5d-6zm9c\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:05.254450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.250760    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb96eb5d-6zm9c\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:05.276530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.270332    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb96eb5d-6zm9c\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:05.308452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.307885    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz87q\" (UniqueName: \"kubernetes.io/projected/be5fb81f-7fe6-49b6-aa63-39d0f161104f-kube-api-access-pz87q\") pod \"maas-api-revoke-sub-eb96eb5d-6zm9c\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:05.329487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.327789    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:05.822680 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:03:05.821889    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5fb81f_7fe6_49b6_aa63_39d0f161104f.slice/crio-19af26eca93624348988b9863d3a009d35f737fcbc158542f700adbe15766499 WatchSource:0}: Error finding container 19af26eca93624348988b9863d3a009d35f737fcbc158542f700adbe15766499: Status 404 returned error can't find the container with id 19af26eca93624348988b9863d3a009d35f737fcbc158542f700adbe15766499
Nov 01 04:03:05.882366 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.880937    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"]
Nov 01 04:03:05.902389 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:05.902350    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c" event={"ID":"be5fb81f-7fe6-49b6-aa63-39d0f161104f","Type":"ContainerStarted","Data":"19af26eca93624348988b9863d3a009d35f737fcbc158542f700adbe15766499"}
Nov 01 04:03:06.679511 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:06.654607    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:06.731452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:06.730550    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:06.764453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:06.759595    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 04:03:06.824587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:06.824531    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:06.984454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:06.973959    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c" event={"ID":"be5fb81f-7fe6-49b6-aa63-39d0f161104f","Type":"ContainerStarted","Data":"fd5f8bd4732ccb574b07973a0ad1a966e94dbafb37be9f0955903351b47d2709"}
Nov 01 04:03:06.990569 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:06.990516    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:07.102565 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.090739    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c" podStartSLOduration=3.09072049 podStartE2EDuration="3.09072049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:07.089884468 +0000 UTC m=+1255.651917486" watchObservedRunningTime="2026-11-01 04:03:07.09072049 +0000 UTC m=+1255.652753510"
Nov 01 04:03:07.161467 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.156101    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:07.367530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.363523    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52" podUID="91fcd874-e74a-4bf5-a6ca-3842296d22b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:07.367530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.364770    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:07.671699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.671600    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:07.737454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.731527    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:07.737454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.731985    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 04:03:07.766552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.766410    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:07.766989 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.766953    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 04:03:07.799453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.796515    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:07.829966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.829922    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:07.830635 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.830608    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 04:03:07.951514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:07.933542    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:08.021454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.010635    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:08.021454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.014600    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:03:08.178567 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.178519    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:08.219556 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.209930    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:03:08.376452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.373673    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:08.377750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.376719    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52" podUID="91fcd874-e74a-4bf5-a6ca-3842296d22b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:08.534648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.465775    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:03:08.534648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.465862    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 04:03:08.534648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.467504    4882 generic.go:356] "Generic (PLEG): container finished" podID="be5fb81f-7fe6-49b6-aa63-39d0f161104f" containerID="fd5f8bd4732ccb574b07973a0ad1a966e94dbafb37be9f0955903351b47d2709" exitCode=0
Nov 01 04:03:08.534648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.467699    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c" event={"ID":"be5fb81f-7fe6-49b6-aa63-39d0f161104f","Type":"ContainerDied","Data":"fd5f8bd4732ccb574b07973a0ad1a966e94dbafb37be9f0955903351b47d2709"}
Nov 01 04:03:08.562453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.543420    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:03:08.562453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.543504    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:08.730379 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.700742    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:08.730379 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.714545    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 04:03:08.746461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.735519    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:08.768523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.768473    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:08.819310 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.817528    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:08.860467 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.819626    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 04:03:08.865586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:08.865525    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 01 04:03:09.212512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.013636    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Nov 01 04:03:09.254453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.254134    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded"
Nov 01 04:03:09.382449 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.379509    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52" podUID="91fcd874-e74a-4bf5-a6ca-3842296d22b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:09.382449 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.380274    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:09.432758 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.432724    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:03:09.433083 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.433049    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:09.749603 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.746287    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 04:03:09.811842 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.761099    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 01 04:03:09.811842 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.769918    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:09.859639 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.859586    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:09.959077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.941685    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": context deadline exceeded"
Nov 01 04:03:09.959077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.954816    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:03:09.978729 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.977979    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:03:09.978729 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.978044    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 04:03:09.978729 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.978086    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52"
Nov 01 04:03:09.978729 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.978115    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 04:03:09.978729 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.978144    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 04:03:09.978729 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.978173    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 04:03:09.995156 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:09.995128    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 04:03:10.015183 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.014508    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded"
Nov 01 04:03:10.072304 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.071866    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:03:10.076571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.075670    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:03:10.272963 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.272871    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"]
Nov 01 04:03:10.283368 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.283339    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.293029 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.292978    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"]
Nov 01 04:03:10.413517 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.391222    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-078e0d65-openshift-default-dockercfg-dr25l\""
Nov 01 04:03:10.562621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.561577    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.562621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.561668    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.562621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.561736    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv7cp\" (UniqueName: \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-kube-api-access-vv7cp\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.562621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.561826    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.562621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.561864    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.562621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.561948    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.562621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.562013    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.562621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.562058    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.562621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.562088    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.619964 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.619932    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:10.663343 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.663293    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-openshift-ca\") pod \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") "
Nov 01 04:03:10.663528 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.663511    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5fb81f-7fe6-49b6-aa63-39d0f161104f-kube-api-access-pz87q\" (UniqueName: \"kubernetes.io/projected/be5fb81f-7fe6-49b6-aa63-39d0f161104f-kube-api-access-pz87q\") pod \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") "
Nov 01 04:03:10.663676 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.663652    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-xks-ca\") pod \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\" (UID: \"be5fb81f-7fe6-49b6-aa63-39d0f161104f\") "
Nov 01 04:03:10.663892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.663682    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-openshift-ca" pod "be5fb81f-7fe6-49b6-aa63-39d0f161104f" (UID: "be5fb81f-7fe6-49b6-aa63-39d0f161104f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:03:10.664237 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.664216    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.664336 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.664296    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.664394 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.664340    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.664394 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.664371    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.664515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.664489    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.664988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.664584    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.664988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.664643    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv7cp\" (UniqueName: \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-kube-api-access-vv7cp\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.664988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.664800    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.664988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.664848    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.664988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.664640    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.665277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.665087    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:03:10.665781 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.665645    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.665972 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.665950    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.666476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.666186    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-xks-ca" pod "be5fb81f-7fe6-49b6-aa63-39d0f161104f" (UID: "be5fb81f-7fe6-49b6-aa63-39d0f161104f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:03:10.666652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.666613    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.666724 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.666669    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5fb81f-7fe6-49b6-aa63-39d0f161104f-kube-api-access-pz87q" pod "be5fb81f-7fe6-49b6-aa63-39d0f161104f" (UID: "be5fb81f-7fe6-49b6-aa63-39d0f161104f"). InnerVolumeSpecName "kube-api-access-pz87q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:03:10.667124 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.667101    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.667579 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.667557    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.668255 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.668232    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.724205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.724170    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv7cp\" (UniqueName: \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-kube-api-access-vv7cp\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.726504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.726471    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:10.766504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.766474    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz87q\" (UniqueName: \"kubernetes.io/projected/be5fb81f-7fe6-49b6-aa63-39d0f161104f-kube-api-access-pz87q\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:03:10.766504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.766503    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5fb81f-7fe6-49b6-aa63-39d0f161104f-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:03:10.865018 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:10.864932    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:11.007188 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:11.007152    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c" event={"ID":"be5fb81f-7fe6-49b6-aa63-39d0f161104f","Type":"ContainerDied","Data":"19af26eca93624348988b9863d3a009d35f737fcbc158542f700adbe15766499"}
Nov 01 04:03:11.007407 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:11.007215    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19af26eca93624348988b9863d3a009d35f737fcbc158542f700adbe15766499"
Nov 01 04:03:11.007407 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:11.007284    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"
Nov 01 04:03:11.362683 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:03:11.362649    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41c7b7c_4552_44f1_b3dd_eb95fe66c466.slice/crio-c0df1e4691e56e5c8302e11ffa9d4681e079031bd59f04c9b575028710cf8a68 WatchSource:0}: Error finding container c0df1e4691e56e5c8302e11ffa9d4681e079031bd59f04c9b575028710cf8a68: Status 404 returned error can't find the container with id c0df1e4691e56e5c8302e11ffa9d4681e079031bd59f04c9b575028710cf8a68
Nov 01 04:03:11.364021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:11.363991    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"]
Nov 01 04:03:11.365378 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:11.365339    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:03:11.365486 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:11.365467    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:03:11.365538 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:11.365520    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:03:12.016048 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:12.016009    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" event={"ID":"e41c7b7c-4552-44f1-b3dd-eb95fe66c466","Type":"ContainerStarted","Data":"ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a"}
Nov 01 04:03:12.016526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:12.016067    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" event={"ID":"e41c7b7c-4552-44f1-b3dd-eb95fe66c466","Type":"ContainerStarted","Data":"c0df1e4691e56e5c8302e11ffa9d4681e079031bd59f04c9b575028710cf8a68"}
Nov 01 04:03:12.131847 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:12.131793    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" podStartSLOduration=2.131775044 podStartE2EDuration="2.131775044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:12.127941736 +0000 UTC m=+1260.689974753" watchObservedRunningTime="2026-11-01 04:03:12.131775044 +0000 UTC m=+1260.693808055"
Nov 01 04:03:12.865369 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:12.865333    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:12.871039 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:12.871015    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:13.026766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:13.026730    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:13.028587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:13.028566    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:03:14.048133 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:14.048099    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"]
Nov 01 04:03:14.094076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:14.094045    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0ba4d5-vlpq4"]
Nov 01 04:03:15.600194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:15.600158    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a35bd3-47bf-4bed-b5c9-7d34bc77d377" path="/var/lib/kubelet/pods/d8a35bd3-47bf-4bed-b5c9-7d34bc77d377/volumes"
Nov 01 04:03:19.034911 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.034878    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"]
Nov 01 04:03:19.035608 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.035587    4882 cpu_manager.go:414] "containerMap: removing container" podUID="be5fb81f-7fe6-49b6-aa63-39d0f161104f" containerName="revoke-keys"
Nov 01 04:03:19.035697 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.035610    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5fb81f-7fe6-49b6-aa63-39d0f161104f" containerName="revoke-keys"
Nov 01 04:03:19.036033 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.035729    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5fb81f-7fe6-49b6-aa63-39d0f161104f" containerName="revoke-keys"
Nov 01 04:03:19.040805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.040784    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.137662 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.137624    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhpr2\" (UniqueName: \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-kube-api-access-qhpr2\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.137856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.137775    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.137856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.137848    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.138002 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.137905    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.138002 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.137921    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.138002 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.137949    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.138151 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.138047    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.138151 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.138091    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.138227 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.138204    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f0d74-fe60-4658-98ec-e133c24cd0fb-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.143260 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.143222    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"]
Nov 01 04:03:19.239838 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.239798    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.239838 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.239853    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240116 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.239917    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240116 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.239933    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240116 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.239957    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240116 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.240002    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240116 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.240049    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240361 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.240166    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f0d74-fe60-4658-98ec-e133c24cd0fb-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240361 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.240212    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhpr2\" (UniqueName: \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-kube-api-access-qhpr2\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240528 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.240367    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240528 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.240370    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240528 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.240447    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240677 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.240601    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.240974 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.240955    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f0d74-fe60-4658-98ec-e133c24cd0fb-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.242508 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.242480    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.242715 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.242694    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.297250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.297174    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.297387 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.297267    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhpr2\" (UniqueName: \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-kube-api-access-qhpr2\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.355375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.355338    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:19.784691 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.784653    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"]
Nov 01 04:03:19.819304 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:03:19.819274    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405f0d74_fe60_4658_98ec_e133c24cd0fb.slice/crio-4772a692431e0222655069d0b43e2a3975c8805dfd9a4698432ddfcf5c64ba6a WatchSource:0}: Error finding container 4772a692431e0222655069d0b43e2a3975c8805dfd9a4698432ddfcf5c64ba6a: Status 404 returned error can't find the container with id 4772a692431e0222655069d0b43e2a3975c8805dfd9a4698432ddfcf5c64ba6a
Nov 01 04:03:19.821628 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.821596    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:03:19.821713 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.821666    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:03:19.821713 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:19.821694    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:03:20.082548 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:20.082446    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" event={"ID":"405f0d74-fe60-4658-98ec-e133c24cd0fb","Type":"ContainerStarted","Data":"d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed"}
Nov 01 04:03:20.082548 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:20.082519    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" event={"ID":"405f0d74-fe60-4658-98ec-e133c24cd0fb","Type":"ContainerStarted","Data":"4772a692431e0222655069d0b43e2a3975c8805dfd9a4698432ddfcf5c64ba6a"}
Nov 01 04:03:20.196894 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:20.196839    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" podStartSLOduration=2.196824516 podStartE2EDuration="2.196824516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:20.195558499 +0000 UTC m=+1268.757591528" watchObservedRunningTime="2026-11-01 04:03:20.196824516 +0000 UTC m=+1268.758857532"
Nov 01 04:03:20.357908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:20.357807    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:20.358248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:20.358224    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 04:03:20.358404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:20.358377    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 04:03:21.355757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.355717    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 04:03:21.356158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.355789    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 04:03:21.588453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.588225    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"]
Nov 01 04:03:21.598815 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.598783    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.634266 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.634232    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"]
Nov 01 04:03:21.691781 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.691746    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-182e51-openshift-default-dockercfg-wxm84\""
Nov 01 04:03:21.699481 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.699451    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-socket\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.700023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.699996    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-istio-token\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.700121 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.700053    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f68cfc4-6033-4820-bb68-e19695d59202-istio-podinfo\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.700121 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.700081    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7khnb\" (UniqueName: \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-kube-api-access-7khnb\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.700121 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.700113    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-credential-socket\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.700286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.700139    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-envoy\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.700286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.700172    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f68cfc4-6033-4820-bb68-e19695d59202-istiod-ca-cert\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.700590 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.700566    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-certs\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.700814 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.700756    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-data\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.815454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.812304    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-istio-token\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.815454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.812406    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f68cfc4-6033-4820-bb68-e19695d59202-istio-podinfo\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.812467    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7khnb\" (UniqueName: \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-kube-api-access-7khnb\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.867929    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-credential-socket\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.867978    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-envoy\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.868043    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f68cfc4-6033-4820-bb68-e19695d59202-istiod-ca-cert\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.868170    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-certs\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.868213    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-data\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.868358    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-socket\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.868937    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-socket\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.869198    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-credential-socket\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.879650    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-certs\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.880322    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f68cfc4-6033-4820-bb68-e19695d59202-istiod-ca-cert\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.881638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.880599    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-data\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.886454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.886144    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-envoy\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.912155 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.909606    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f68cfc4-6033-4820-bb68-e19695d59202-istio-podinfo\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.913678 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.913649    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-istio-token\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:21.920770 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:21.920740    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7khnb\" (UniqueName: \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-kube-api-access-7khnb\") pod \"e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") " pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:22.047337 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:22.047158    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"]
Nov 01 04:03:22.051832 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:22.051791    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bb3559-ps8dr"]
Nov 01 04:03:22.218135 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:22.218106    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:22.356560 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:22.356152    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 04:03:22.356560 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:22.356208    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 04:03:22.691882 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:03:22.691837    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f68cfc4_6033_4820_bb68_e19695d59202.slice/crio-201d2272e6085c033978d464841b5e84eae79337ffa6de058502c06ccc6ad6f1 WatchSource:0}: Error finding container 201d2272e6085c033978d464841b5e84eae79337ffa6de058502c06ccc6ad6f1: Status 404 returned error can't find the container with id 201d2272e6085c033978d464841b5e84eae79337ffa6de058502c06ccc6ad6f1
Nov 01 04:03:22.692230 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:22.692208    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"]
Nov 01 04:03:22.694439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:22.694387    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:03:22.694544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:22.694482    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:03:22.694544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:22.694530    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:03:23.114272 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:23.114172    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" event={"ID":"4f68cfc4-6033-4820-bb68-e19695d59202","Type":"ContainerStarted","Data":"9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191"}
Nov 01 04:03:23.114272 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:23.114242    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" event={"ID":"4f68cfc4-6033-4820-bb68-e19695d59202","Type":"ContainerStarted","Data":"201d2272e6085c033978d464841b5e84eae79337ffa6de058502c06ccc6ad6f1"}
Nov 01 04:03:23.218281 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:23.218250    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:23.223369 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:23.223343    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:23.233023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:23.232974    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" podStartSLOduration=2.232960945 podStartE2EDuration="2.232960945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:23.230393554 +0000 UTC m=+1271.792426571" watchObservedRunningTime="2026-11-01 04:03:23.232960945 +0000 UTC m=+1271.794993961"
Nov 01 04:03:23.358033 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:23.357985    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 04:03:23.358576 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:23.358047    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 04:03:23.600406 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:23.600372    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3e197f-148e-4fa0-a54e-081112abbebe" path="/var/lib/kubelet/pods/1d3e197f-148e-4fa0-a54e-081112abbebe/volumes"
Nov 01 04:03:24.121971 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.121935    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:24.122854 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.122835    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:03:24.356695 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.356660    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 04:03:24.356837 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.356720    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 04:03:24.882951 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.882905    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"]
Nov 01 04:03:24.890666 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.890633    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:24.921344 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.921305    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"]
Nov 01 04:03:24.926222 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.926189    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsct6\" (UniqueName: \"kubernetes.io/projected/120956bd-4b22-467c-a6c5-56ea377c9c7d-kube-api-access-qsct6\") pod \"maas-api-revoke-sub-0d98b25e-m4dpm\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:24.926372 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.926349    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d98b25e-m4dpm\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:24.926584 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.926562    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d98b25e-m4dpm\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:24.939283 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.939261    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:03:24.939421 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:24.939297    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:03:25.028372 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:25.028336    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d98b25e-m4dpm\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:25.028557 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:25.028496    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d98b25e-m4dpm\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:25.028607 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:25.028565    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsct6\" (UniqueName: \"kubernetes.io/projected/120956bd-4b22-467c-a6c5-56ea377c9c7d-kube-api-access-qsct6\") pod \"maas-api-revoke-sub-0d98b25e-m4dpm\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:25.029102 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:25.029081    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d98b25e-m4dpm\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:25.030995 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:25.030973    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d98b25e-m4dpm\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:25.088758 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:25.088723    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsct6\" (UniqueName: \"kubernetes.io/projected/120956bd-4b22-467c-a6c5-56ea377c9c7d-kube-api-access-qsct6\") pod \"maas-api-revoke-sub-0d98b25e-m4dpm\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:25.206219 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:25.206177    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:25.356109 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:25.356070    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 04:03:25.356304 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:25.356135    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 04:03:25.609514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:25.609487    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"]
Nov 01 04:03:25.609786 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:03:25.609761    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120956bd_4b22_467c_a6c5_56ea377c9c7d.slice/crio-4d997af9c179a5f6f2e0b19d1273399bd05d7f69204094d8389f3a930c3d951e WatchSource:0}: Error finding container 4d997af9c179a5f6f2e0b19d1273399bd05d7f69204094d8389f3a930c3d951e: Status 404 returned error can't find the container with id 4d997af9c179a5f6f2e0b19d1273399bd05d7f69204094d8389f3a930c3d951e
Nov 01 04:03:26.137772 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:26.137732    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm" event={"ID":"120956bd-4b22-467c-a6c5-56ea377c9c7d","Type":"ContainerStarted","Data":"0d1cef3c63cc8cff4d7c23bdd1d9fc2694e4e187be4b5a9016ddd636c0dbba23"}
Nov 01 04:03:26.138228 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:26.137804    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm" event={"ID":"120956bd-4b22-467c-a6c5-56ea377c9c7d","Type":"ContainerStarted","Data":"4d997af9c179a5f6f2e0b19d1273399bd05d7f69204094d8389f3a930c3d951e"}
Nov 01 04:03:26.253671 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:26.253575    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm" podStartSLOduration=2.253562182 podStartE2EDuration="2.253562182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:26.250684504 +0000 UTC m=+1274.812717521" watchObservedRunningTime="2026-11-01 04:03:26.253562182 +0000 UTC m=+1274.815595234"
Nov 01 04:03:26.356517 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:26.356485    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 01 04:03:26.356662 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:26.356536    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 01 04:03:27.149351 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:27.149314    4882 generic.go:356] "Generic (PLEG): container finished" podID="120956bd-4b22-467c-a6c5-56ea377c9c7d" containerID="0d1cef3c63cc8cff4d7c23bdd1d9fc2694e4e187be4b5a9016ddd636c0dbba23" exitCode=0
Nov 01 04:03:27.149802 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:27.149408    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm" event={"ID":"120956bd-4b22-467c-a6c5-56ea377c9c7d","Type":"ContainerDied","Data":"0d1cef3c63cc8cff4d7c23bdd1d9fc2694e4e187be4b5a9016ddd636c0dbba23"}
Nov 01 04:03:27.360387 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:27.360338    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:27.360910 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:27.360888    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:27.361550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:27.361531    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:03:28.331914 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:28.331887    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:28.374799 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:28.374766    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-openshift-ca\") pod \"120956bd-4b22-467c-a6c5-56ea377c9c7d\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") "
Nov 01 04:03:28.374978 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:28.374823    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/120956bd-4b22-467c-a6c5-56ea377c9c7d-kube-api-access-qsct6\" (UniqueName: \"kubernetes.io/projected/120956bd-4b22-467c-a6c5-56ea377c9c7d-kube-api-access-qsct6\") pod \"120956bd-4b22-467c-a6c5-56ea377c9c7d\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") "
Nov 01 04:03:28.374978 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:28.374848    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-xks-ca\") pod \"120956bd-4b22-467c-a6c5-56ea377c9c7d\" (UID: \"120956bd-4b22-467c-a6c5-56ea377c9c7d\") "
Nov 01 04:03:28.375155 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:28.375133    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-openshift-ca" pod "120956bd-4b22-467c-a6c5-56ea377c9c7d" (UID: "120956bd-4b22-467c-a6c5-56ea377c9c7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:03:28.375447 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:28.375413    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:03:28.377003 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:28.376981    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/120956bd-4b22-467c-a6c5-56ea377c9c7d-kube-api-access-qsct6" pod "120956bd-4b22-467c-a6c5-56ea377c9c7d" (UID: "120956bd-4b22-467c-a6c5-56ea377c9c7d"). InnerVolumeSpecName "kube-api-access-qsct6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:03:28.377082 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:28.377067    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-xks-ca" pod "120956bd-4b22-467c-a6c5-56ea377c9c7d" (UID: "120956bd-4b22-467c-a6c5-56ea377c9c7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:03:28.477279 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:28.477237    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsct6\" (UniqueName: \"kubernetes.io/projected/120956bd-4b22-467c-a6c5-56ea377c9c7d-kube-api-access-qsct6\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:03:28.477279 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:28.477282    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/120956bd-4b22-467c-a6c5-56ea377c9c7d-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:03:28.726254 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:03:28.726218    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:03:29.166690 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:29.166658    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"
Nov 01 04:03:29.166921 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:29.166698    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm" event={"ID":"120956bd-4b22-467c-a6c5-56ea377c9c7d","Type":"ContainerDied","Data":"4d997af9c179a5f6f2e0b19d1273399bd05d7f69204094d8389f3a930c3d951e"}
Nov 01 04:03:29.166921 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:29.166748    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d997af9c179a5f6f2e0b19d1273399bd05d7f69204094d8389f3a930c3d951e"
Nov 01 04:03:33.051927 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:33.051883    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"]
Nov 01 04:03:33.059168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:33.059133    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9751f1-mq6vt"]
Nov 01 04:03:33.644376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:33.644333    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69778b9-3a55-4747-b34a-056c75dc57f5" path="/var/lib/kubelet/pods/e69778b9-3a55-4747-b34a-056c75dc57f5/volumes"
Nov 01 04:03:49.616659 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.616621    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"]
Nov 01 04:03:49.617541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.617287    4882 cpu_manager.go:414] "containerMap: removing container" podUID="120956bd-4b22-467c-a6c5-56ea377c9c7d" containerName="revoke-keys"
Nov 01 04:03:49.617541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.617303    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="120956bd-4b22-467c-a6c5-56ea377c9c7d" containerName="revoke-keys"
Nov 01 04:03:49.617541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.617451    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="120956bd-4b22-467c-a6c5-56ea377c9c7d" containerName="revoke-keys"
Nov 01 04:03:49.626035 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.626003    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.676284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.676242    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 04:03:49.676490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.676308    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 04:03:49.679220 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.679188    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"]
Nov 01 04:03:49.716587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.716540    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.716587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.716592    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.716849 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.716695    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.717064 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.717037    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595bk\" (UniqueName: \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-kube-api-access-595bk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.717225 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.717197    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.717305 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.717288    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6b22ce-68d5-4f70-9378-62d19b20c73c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.717364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.717330    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.719575 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.719554    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5gxzd\""
Nov 01 04:03:49.719575 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.719572    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 04:03:49.819208 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.819173    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.819404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.819239    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.819404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.819283    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.819404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.819337    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595bk\" (UniqueName: \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-kube-api-access-595bk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.819404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.819377    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.819673 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.819415    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6b22ce-68d5-4f70-9378-62d19b20c73c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.819673 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.819509    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.820274 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.820244    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.821939 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.821911    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.822286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.822262    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.822468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.822416    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.823382 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.823361    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6b22ce-68d5-4f70-9378-62d19b20c73c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.877879 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.877792    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-595bk\" (UniqueName: \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-kube-api-access-595bk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.877879 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.877858    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k295k\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:49.949594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:49.949561    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:50.392479 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:03:50.392410    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6b22ce_68d5_4f70_9378_62d19b20c73c.slice/crio-74a1d8fe74a49e693f4bd76405f980f6ed071269282dd8ba743346bc606c2e59 WatchSource:0}: Error finding container 74a1d8fe74a49e693f4bd76405f980f6ed071269282dd8ba743346bc606c2e59: Status 404 returned error can't find the container with id 74a1d8fe74a49e693f4bd76405f980f6ed071269282dd8ba743346bc606c2e59
Nov 01 04:03:50.393583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:50.393550    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"]
Nov 01 04:03:50.394688 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:50.394672    4882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 04:03:50.409491 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:50.409456    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k" event={"ID":"3c6b22ce-68d5-4f70-9378-62d19b20c73c","Type":"ContainerStarted","Data":"74a1d8fe74a49e693f4bd76405f980f6ed071269282dd8ba743346bc606c2e59"}
Nov 01 04:03:52.344768 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.344732    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"]
Nov 01 04:03:52.350973 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.350931    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:52.402902 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.402870    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"]
Nov 01 04:03:52.422531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.422486    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:03:52.422718 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.422640    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:03:52.461149 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.461115    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-705c0aa1-66rf2\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:52.461149 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.461151    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-705c0aa1-66rf2\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:52.461349 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.461287    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gxl6\" (UniqueName: \"kubernetes.io/projected/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-kube-api-access-2gxl6\") pod \"maas-api-revoke-sub-705c0aa1-66rf2\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:52.562411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.562375    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-705c0aa1-66rf2\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:52.562614 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.562418    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-705c0aa1-66rf2\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:52.562614 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.562533    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gxl6\" (UniqueName: \"kubernetes.io/projected/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-kube-api-access-2gxl6\") pod \"maas-api-revoke-sub-705c0aa1-66rf2\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:52.563096 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.563074    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-705c0aa1-66rf2\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:52.564821 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.564803    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-705c0aa1-66rf2\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:52.622199 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.622113    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gxl6\" (UniqueName: \"kubernetes.io/projected/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-kube-api-access-2gxl6\") pod \"maas-api-revoke-sub-705c0aa1-66rf2\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:52.670886 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:52.670840    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:53.061843 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:53.061808    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"]
Nov 01 04:03:53.062712 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:03:53.062681    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6be760_8e8c_46d0_a7d4_877e3660f9d0.slice/crio-fad09d1927a93f0af5483cea5d301aecc7117f300a4ac1e401edc0507c133fd3 WatchSource:0}: Error finding container fad09d1927a93f0af5483cea5d301aecc7117f300a4ac1e401edc0507c133fd3: Status 404 returned error can't find the container with id fad09d1927a93f0af5483cea5d301aecc7117f300a4ac1e401edc0507c133fd3
Nov 01 04:03:53.448753 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:53.448721    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2" event={"ID":"bb6be760-8e8c-46d0-a7d4-877e3660f9d0","Type":"ContainerStarted","Data":"610d888314db586389efa3c7cae57496d793754e141031fbebc4f26f54e9e045"}
Nov 01 04:03:53.449176 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:53.448771    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2" event={"ID":"bb6be760-8e8c-46d0-a7d4-877e3660f9d0","Type":"ContainerStarted","Data":"fad09d1927a93f0af5483cea5d301aecc7117f300a4ac1e401edc0507c133fd3"}
Nov 01 04:03:53.560296 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:53.560243    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2" podStartSLOduration=1.560230043 podStartE2EDuration="1.560230043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:53.556766697 +0000 UTC m=+1302.118799715" watchObservedRunningTime="2026-11-01 04:03:53.560230043 +0000 UTC m=+1302.122263060"
Nov 01 04:03:54.457402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:54.457360    4882 generic.go:356] "Generic (PLEG): container finished" podID="bb6be760-8e8c-46d0-a7d4-877e3660f9d0" containerID="610d888314db586389efa3c7cae57496d793754e141031fbebc4f26f54e9e045" exitCode=0
Nov 01 04:03:54.457853 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:54.457452    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2" event={"ID":"bb6be760-8e8c-46d0-a7d4-877e3660f9d0","Type":"ContainerDied","Data":"610d888314db586389efa3c7cae57496d793754e141031fbebc4f26f54e9e045"}
Nov 01 04:03:55.652219 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:55.652189    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:55.816388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:55.816289    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-kube-api-access-2gxl6\" (UniqueName: \"kubernetes.io/projected/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-kube-api-access-2gxl6\") pod \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") "
Nov 01 04:03:55.816388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:55.816348    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-xks-ca\") pod \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") "
Nov 01 04:03:55.816668 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:55.816591    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-openshift-ca\") pod \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\" (UID: \"bb6be760-8e8c-46d0-a7d4-877e3660f9d0\") "
Nov 01 04:03:55.817144 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:55.817112    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-openshift-ca" pod "bb6be760-8e8c-46d0-a7d4-877e3660f9d0" (UID: "bb6be760-8e8c-46d0-a7d4-877e3660f9d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:03:55.819081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:55.819047    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-xks-ca" pod "bb6be760-8e8c-46d0-a7d4-877e3660f9d0" (UID: "bb6be760-8e8c-46d0-a7d4-877e3660f9d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:03:55.819208 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:55.819124    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-kube-api-access-2gxl6" pod "bb6be760-8e8c-46d0-a7d4-877e3660f9d0" (UID: "bb6be760-8e8c-46d0-a7d4-877e3660f9d0"). InnerVolumeSpecName "kube-api-access-2gxl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:03:55.918956 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:55.918914    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:03:55.918956 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:55.918949    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gxl6\" (UniqueName: \"kubernetes.io/projected/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-kube-api-access-2gxl6\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:03:55.918956 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:55.918959    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6be760-8e8c-46d0-a7d4-877e3660f9d0-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:03:56.478383 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:56.478347    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"
Nov 01 04:03:56.478383 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:56.478378    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2" event={"ID":"bb6be760-8e8c-46d0-a7d4-877e3660f9d0","Type":"ContainerDied","Data":"fad09d1927a93f0af5483cea5d301aecc7117f300a4ac1e401edc0507c133fd3"}
Nov 01 04:03:56.478675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:56.478460    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad09d1927a93f0af5483cea5d301aecc7117f300a4ac1e401edc0507c133fd3"
Nov 01 04:03:58.131498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:58.131454    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:03:58.131823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:58.131550    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:03:58.502337 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:58.502251    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k" event={"ID":"3c6b22ce-68d5-4f70-9378-62d19b20c73c","Type":"ContainerStarted","Data":"8c3ff016ae3e6b7fe1f467a885925c0ea69d6c6df18f61ca6a6300d7f342a6e4"}
Nov 01 04:03:58.502597 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:58.502362    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:03:58.667009 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:58.666951    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k" podStartSLOduration=1.931884035 podStartE2EDuration="9.666934648s" totalImagesPullingTime="7.735050613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:58.663807523 +0000 UTC m=+1307.225840540" watchObservedRunningTime="2026-11-01 04:03:58.666934648 +0000 UTC m=+1307.228967664"
Nov 01 04:03:58.729704 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:03:58.729650    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:03:59.513817 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:59.513758    4882 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-k295k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:03:59.514330 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:03:59.513838    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k" podUID="3c6b22ce-68d5-4f70-9378-62d19b20c73c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:04:02.512406 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:02.512376    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:04:12.160641 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:12.160609    4882 scope.go:122] "RemoveContainer" containerID="29b05b62507662b2eb0a7adccc5e4bda76a43858ee968e369ce8f8c0f15f76a4"
Nov 01 04:04:12.174289 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:12.174178    4882 scope.go:122] "RemoveContainer" containerID="afede98c939c3d38671842c784e757eeb366c59ce359d882bf450f867cb07231"
Nov 01 04:04:12.184438 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:12.184407    4882 scope.go:122] "RemoveContainer" containerID="0c55af4ffce59a231874ca638962b8aaa08d027068ba6a1cccc9c7ea0474a9da"
Nov 01 04:04:12.194324 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:12.194301    4882 scope.go:122] "RemoveContainer" containerID="2a080ab54b2057261177968c922d81b112c2411ead11239ec13be23f7a8a7d7d"
Nov 01 04:04:12.203617 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:12.203598    4882 scope.go:122] "RemoveContainer" containerID="0481eb0ba71faa948630466f2afe90813e723f01427c042e99c783a4feef9ce0"
Nov 01 04:04:12.212379 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:12.212360    4882 scope.go:122] "RemoveContainer" containerID="2c16c45267878feecc26b2dac7f3385fb8f6547bda790ef22c5e1d9d54191975"
Nov 01 04:04:28.723564 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:04:28.723523    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:04:40.168666 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:40.168610    4882 patch_prober.go:29] interesting pod/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:04:40.169168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:40.168714    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:04:41.069340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.069300    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"]
Nov 01 04:04:41.070126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.070102    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bb6be760-8e8c-46d0-a7d4-877e3660f9d0" containerName="revoke-keys"
Nov 01 04:04:41.070126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.070122    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6be760-8e8c-46d0-a7d4-877e3660f9d0" containerName="revoke-keys"
Nov 01 04:04:41.070346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.070240    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6be760-8e8c-46d0-a7d4-877e3660f9d0" containerName="revoke-keys"
Nov 01 04:04:41.088563 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.088536    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:41.126919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.126629    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8cb6dac-bwrkv\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:41.126919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.126709    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8cb6dac-bwrkv\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:41.126919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.126758    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8mzk\" (UniqueName: \"kubernetes.io/projected/004ac357-ceb3-499a-903c-1204fce4a504-kube-api-access-x8mzk\") pod \"maas-api-revoke-sub-e8cb6dac-bwrkv\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:41.141764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.141473    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:04:41.142498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.142472    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:04:41.154853 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.154785    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"]
Nov 01 04:04:41.230786 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.230734    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8cb6dac-bwrkv\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:41.231320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.230825    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8cb6dac-bwrkv\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:41.231320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.230873    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8mzk\" (UniqueName: \"kubernetes.io/projected/004ac357-ceb3-499a-903c-1204fce4a504-kube-api-access-x8mzk\") pod \"maas-api-revoke-sub-e8cb6dac-bwrkv\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:41.232645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.232034    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8cb6dac-bwrkv\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:41.250686 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.248524    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8cb6dac-bwrkv\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:41.290691 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.290655    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8mzk\" (UniqueName: \"kubernetes.io/projected/004ac357-ceb3-499a-903c-1204fce4a504-kube-api-access-x8mzk\") pod \"maas-api-revoke-sub-e8cb6dac-bwrkv\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:41.426008 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.425909    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:41.833774 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:04:41.833740    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004ac357_ceb3_499a_903c_1204fce4a504.slice/crio-bf6922654bc4a782d7551b663c7dced9a17bad7fc311bf029926f14c128f8866 WatchSource:0}: Error finding container bf6922654bc4a782d7551b663c7dced9a17bad7fc311bf029926f14c128f8866: Status 404 returned error can't find the container with id bf6922654bc4a782d7551b663c7dced9a17bad7fc311bf029926f14c128f8866
Nov 01 04:04:41.834485 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:41.834386    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"]
Nov 01 04:04:42.064402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:42.064356    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv" event={"ID":"004ac357-ceb3-499a-903c-1204fce4a504","Type":"ContainerStarted","Data":"bf6922654bc4a782d7551b663c7dced9a17bad7fc311bf029926f14c128f8866"}
Nov 01 04:04:43.074025 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:43.073989    4882 generic.go:356] "Generic (PLEG): container finished" podID="004ac357-ceb3-499a-903c-1204fce4a504" containerID="21105a863ed20a73d6689c8fbd678a144521d1e8870f3b534ff063b110ee524f" exitCode=0
Nov 01 04:04:43.074598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:43.074074    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv" event={"ID":"004ac357-ceb3-499a-903c-1204fce4a504","Type":"ContainerDied","Data":"21105a863ed20a73d6689c8fbd678a144521d1e8870f3b534ff063b110ee524f"}
Nov 01 04:04:43.187158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:43.185099    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": read tcp 10.132.0.2:57036->10.132.0.122:15021: read: connection reset by peer" start-of-body=
Nov 01 04:04:43.187158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:43.185198    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": read tcp 10.132.0.2:57036->10.132.0.122:15021: read: connection reset by peer"
Nov 01 04:04:44.083055 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.083021    4882 generic.go:356] "Generic (PLEG): container finished" podID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerID="ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a" exitCode=0
Nov 01 04:04:44.083567 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.083094    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" event={"ID":"e41c7b7c-4552-44f1-b3dd-eb95fe66c466","Type":"ContainerDied","Data":"ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a"}
Nov 01 04:04:44.083649 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.083630    4882 scope.go:122] "RemoveContainer" containerID="ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a"
Nov 01 04:04:44.084932 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.084907    4882 generic.go:356] "Generic (PLEG): container finished" podID="4f68cfc4-6033-4820-bb68-e19695d59202" containerID="9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191" exitCode=0
Nov 01 04:04:44.085057 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.084995    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" event={"ID":"4f68cfc4-6033-4820-bb68-e19695d59202","Type":"ContainerDied","Data":"9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191"}
Nov 01 04:04:44.085346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.085317    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:04:44.085400 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.085382    4882 scope.go:122] "RemoveContainer" containerID="9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191"
Nov 01 04:04:44.085468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.085403    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:04:44.085520 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.085487    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:04:44.087082 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.087040    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:04:44.087181 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.087127    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:04:44.087181 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.087170    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:04:44.364747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.364658    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:44.549216 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.549181    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004ac357-ceb3-499a-903c-1204fce4a504-kube-api-access-x8mzk\" (UniqueName: \"kubernetes.io/projected/004ac357-ceb3-499a-903c-1204fce4a504-kube-api-access-x8mzk\") pod \"004ac357-ceb3-499a-903c-1204fce4a504\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") "
Nov 01 04:04:44.549411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.549266    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-openshift-ca\") pod \"004ac357-ceb3-499a-903c-1204fce4a504\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") "
Nov 01 04:04:44.549522 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.549480    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-xks-ca\") pod \"004ac357-ceb3-499a-903c-1204fce4a504\" (UID: \"004ac357-ceb3-499a-903c-1204fce4a504\") "
Nov 01 04:04:44.552228 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.552195    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-openshift-ca" pod "004ac357-ceb3-499a-903c-1204fce4a504" (UID: "004ac357-ceb3-499a-903c-1204fce4a504"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:04:44.553923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.553897    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-xks-ca" pod "004ac357-ceb3-499a-903c-1204fce4a504" (UID: "004ac357-ceb3-499a-903c-1204fce4a504"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:04:44.556501 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.556037    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004ac357-ceb3-499a-903c-1204fce4a504-kube-api-access-x8mzk" pod "004ac357-ceb3-499a-903c-1204fce4a504" (UID: "004ac357-ceb3-499a-903c-1204fce4a504"). InnerVolumeSpecName "kube-api-access-x8mzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:04:44.652864 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.652776    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:04:44.652864 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.652820    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/004ac357-ceb3-499a-903c-1204fce4a504-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:04:44.652864 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:44.652839    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8mzk\" (UniqueName: \"kubernetes.io/projected/004ac357-ceb3-499a-903c-1204fce4a504-kube-api-access-x8mzk\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:04:45.099465 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:45.099403    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"
Nov 01 04:04:45.099465 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:45.099446    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv" event={"ID":"004ac357-ceb3-499a-903c-1204fce4a504","Type":"ContainerDied","Data":"bf6922654bc4a782d7551b663c7dced9a17bad7fc311bf029926f14c128f8866"}
Nov 01 04:04:45.100023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:45.099510    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf6922654bc4a782d7551b663c7dced9a17bad7fc311bf029926f14c128f8866"
Nov 01 04:04:45.101783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:45.101756    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" event={"ID":"4f68cfc4-6033-4820-bb68-e19695d59202","Type":"ContainerStarted","Data":"873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5"}
Nov 01 04:04:45.104086 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:45.104060    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" event={"ID":"e41c7b7c-4552-44f1-b3dd-eb95fe66c466","Type":"ContainerStarted","Data":"f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba"}
Nov 01 04:04:45.219782 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:45.219743    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:04:45.220013 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:45.219842    4882 patch_prober.go:29] interesting pod/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 04:04:45.220013 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:45.219887    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 04:04:45.879451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:45.866806    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:04:45.879451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:45.866931    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 04:04:45.879451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:45.866985    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 04:04:46.219120 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:46.219088    4882 patch_prober.go:29] interesting pod/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 04:04:46.219551 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:46.219148    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 04:04:46.877454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:46.870814    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 04:04:46.877454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:46.870882    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 04:04:47.219165 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:47.219127    4882 patch_prober.go:29] interesting pod/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 04:04:47.219606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:47.219187    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 04:04:47.869454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:47.866385    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 04:04:47.869454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:47.866468    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 04:04:48.219009 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:48.218974    4882 patch_prober.go:29] interesting pod/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 04:04:48.219201 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:48.219037    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 04:04:48.882911 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:48.882865    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 04:04:48.884364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:48.884324    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 04:04:49.219644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:49.219604    4882 patch_prober.go:29] interesting pod/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 04:04:49.220062 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:49.219665    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 04:04:49.867834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:49.867791    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 04:04:49.868159 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:49.868127    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 04:04:50.218993 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:50.218956    4882 patch_prober.go:29] interesting pod/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 01 04:04:50.219183 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:50.219026    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 01 04:04:50.865994 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:50.865948    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 04:04:50.866422 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:50.866028    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 04:04:51.225688 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:51.225656    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:04:51.226109 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:51.226074    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:04:51.227182 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:51.227161    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:04:51.870233 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:51.870202    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:04:51.870747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:51.870644    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:04:51.871534 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:51.871511    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:04:56.717377 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.717340    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"]
Nov 01 04:04:56.718095 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.718069    4882 cpu_manager.go:414] "containerMap: removing container" podUID="004ac357-ceb3-499a-903c-1204fce4a504" containerName="revoke-keys"
Nov 01 04:04:56.718171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.718143    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004ac357-ceb3-499a-903c-1204fce4a504" containerName="revoke-keys"
Nov 01 04:04:56.718404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.718388    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="004ac357-ceb3-499a-903c-1204fce4a504" containerName="revoke-keys"
Nov 01 04:04:56.726740 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.726703    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:04:56.775144 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.775108    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"]
Nov 01 04:04:56.812636 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.812601    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:04:56.812636 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.812632    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:04:56.837618 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.837574    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dc88a5-t42t5\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:04:56.837779 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.837656    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5h8m\" (UniqueName: \"kubernetes.io/projected/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-kube-api-access-f5h8m\") pod \"maas-api-revoke-sub-77dc88a5-t42t5\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:04:56.837866 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.837816    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dc88a5-t42t5\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:04:56.939641 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.939603    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dc88a5-t42t5\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:04:56.939819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.939672    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5h8m\" (UniqueName: \"kubernetes.io/projected/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-kube-api-access-f5h8m\") pod \"maas-api-revoke-sub-77dc88a5-t42t5\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:04:56.939819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.939740    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dc88a5-t42t5\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:04:56.940489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.940423    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dc88a5-t42t5\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:04:56.941993 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.941974    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dc88a5-t42t5\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:04:56.996912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:56.996844    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5h8m\" (UniqueName: \"kubernetes.io/projected/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-kube-api-access-f5h8m\") pod \"maas-api-revoke-sub-77dc88a5-t42t5\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:04:57.063807 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:57.063771    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:04:57.441814 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:57.441773    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"]
Nov 01 04:04:57.443060 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:04:57.443032    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5f4e3a_8889_4e9d_88b9_01e1c7ffb348.slice/crio-49ec1d4f541d2e96df406b72844c324b00b665ff30e5c14819bf1c479126fbd2 WatchSource:0}: Error finding container 49ec1d4f541d2e96df406b72844c324b00b665ff30e5c14819bf1c479126fbd2: Status 404 returned error can't find the container with id 49ec1d4f541d2e96df406b72844c324b00b665ff30e5c14819bf1c479126fbd2
Nov 01 04:04:58.213478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:58.213410    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5" event={"ID":"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348","Type":"ContainerStarted","Data":"eae55469bfdfa81142e88f5ef0263bf5398d8e01c0065979b965ae145521a65d"}
Nov 01 04:04:58.213899 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:58.213508    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5" event={"ID":"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348","Type":"ContainerStarted","Data":"49ec1d4f541d2e96df406b72844c324b00b665ff30e5c14819bf1c479126fbd2"}
Nov 01 04:04:58.323752 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:58.323687    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5" podStartSLOduration=2.3236727950000002 podStartE2EDuration="2.323672795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:04:58.321348795 +0000 UTC m=+1366.883381812" watchObservedRunningTime="2026-11-01 04:04:58.323672795 +0000 UTC m=+1366.885705812"
Nov 01 04:04:58.728342 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:04:58.728297    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:04:59.223064 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:59.223021    4882 generic.go:356] "Generic (PLEG): container finished" podID="7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348" containerID="eae55469bfdfa81142e88f5ef0263bf5398d8e01c0065979b965ae145521a65d" exitCode=0
Nov 01 04:04:59.223456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:04:59.223102    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5" event={"ID":"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348","Type":"ContainerDied","Data":"eae55469bfdfa81142e88f5ef0263bf5398d8e01c0065979b965ae145521a65d"}
Nov 01 04:05:00.400613 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:00.400580    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:05:00.489209 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:00.489120    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-kube-api-access-f5h8m\" (UniqueName: \"kubernetes.io/projected/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-kube-api-access-f5h8m\") pod \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") "
Nov 01 04:05:00.489414 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:00.489401    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-openshift-ca\") pod \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") "
Nov 01 04:05:00.489515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:00.489459    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-xks-ca\") pod \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\" (UID: \"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348\") "
Nov 01 04:05:00.489788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:00.489760    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-openshift-ca" pod "7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348" (UID: "7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:05:00.489969 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:00.489947    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:05:00.491343 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:00.491319    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-kube-api-access-f5h8m" pod "7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348" (UID: "7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348"). InnerVolumeSpecName "kube-api-access-f5h8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:05:00.491554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:00.491535    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-xks-ca" pod "7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348" (UID: "7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:05:00.591751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:00.591715    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:05:00.591751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:00.591747    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5h8m\" (UniqueName: \"kubernetes.io/projected/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348-kube-api-access-f5h8m\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:05:01.241519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:01.241475    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5" event={"ID":"7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348","Type":"ContainerDied","Data":"49ec1d4f541d2e96df406b72844c324b00b665ff30e5c14819bf1c479126fbd2"}
Nov 01 04:05:01.241816 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:01.241788    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49ec1d4f541d2e96df406b72844c324b00b665ff30e5c14819bf1c479126fbd2"
Nov 01 04:05:01.242040 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:01.242005    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"
Nov 01 04:05:12.264379 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:12.264346    4882 scope.go:122] "RemoveContainer" containerID="757246b3f9367ae65745de0eb897c8bea3eaa7164f6735af802901b58908bd87"
Nov 01 04:05:12.276042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:12.276020    4882 scope.go:122] "RemoveContainer" containerID="6a55a87925dcbae7d028b695c7e7cfac6115b90fa890f0ce481bd8f332afd316"
Nov 01 04:05:22.984330 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:22.984239    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"]
Nov 01 04:05:22.985172 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:22.985150    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348" containerName="revoke-keys"
Nov 01 04:05:22.985274 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:22.985172    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348" containerName="revoke-keys"
Nov 01 04:05:22.985340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:22.985325    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348" containerName="revoke-keys"
Nov 01 04:05:22.990355 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:22.990332    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.085338 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.085301    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-credential-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.085686 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.085654    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-data\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.085818 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.085781    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-podinfo\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.085917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.085836    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-token\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.085917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.085853    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"]
Nov 01 04:05:23.085917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.085865    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.085917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.085904    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj2z5\" (UniqueName: \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-kube-api-access-lj2z5\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.086153 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.086126    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-envoy\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.086239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.086224    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1421745-fcd8-41fc-ba64-ff1336e43684-istiod-ca-cert\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.086327 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.086314    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-certs\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.187938 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.187901    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-credential-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188141 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.187969    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-data\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188141 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188004    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-podinfo\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188141 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188023    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-token\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188141 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188042    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188141 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188071    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj2z5\" (UniqueName: \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-kube-api-access-lj2z5\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188172    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-envoy\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188240    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1421745-fcd8-41fc-ba64-ff1336e43684-istiod-ca-cert\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188304    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-certs\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188404    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-credential-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188649 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188455    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-data\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188708 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188693    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.188776 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188754    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-certs\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.189017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.188998    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1421745-fcd8-41fc-ba64-ff1336e43684-istiod-ca-cert\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.190365 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.190343    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-envoy\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.190484 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.190468    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-podinfo\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.245295 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.245231    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-token\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.245522 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.245497    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj2z5\" (UniqueName: \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-kube-api-access-lj2z5\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-2lt96\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.305038 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.305005    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:23.729481 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.729447    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"]
Nov 01 04:05:23.730127 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:05:23.730098    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1421745_fcd8_41fc_ba64_ff1336e43684.slice/crio-e00fd6754e4c72114f3885002cf4178a120b5b11fe3706a84413db316fe8f382 WatchSource:0}: Error finding container e00fd6754e4c72114f3885002cf4178a120b5b11fe3706a84413db316fe8f382: Status 404 returned error can't find the container with id e00fd6754e4c72114f3885002cf4178a120b5b11fe3706a84413db316fe8f382
Nov 01 04:05:23.732809 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.732777    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:05:23.732870 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.732844    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:05:23.732913 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:23.732874    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:05:24.494453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:24.494035    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" event={"ID":"f1421745-fcd8-41fc-ba64-ff1336e43684","Type":"ContainerStarted","Data":"5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb"}
Nov 01 04:05:24.494453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:24.494104    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" event={"ID":"f1421745-fcd8-41fc-ba64-ff1336e43684","Type":"ContainerStarted","Data":"e00fd6754e4c72114f3885002cf4178a120b5b11fe3706a84413db316fe8f382"}
Nov 01 04:05:24.701582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:24.701525    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" podStartSLOduration=2.701511442 podStartE2EDuration="2.701511442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:05:24.653116776 +0000 UTC m=+1393.215149792" watchObservedRunningTime="2026-11-01 04:05:24.701511442 +0000 UTC m=+1393.263544453"
Nov 01 04:05:25.305356 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:25.305320    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:25.306845 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:25.306815    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 04:05:25.306977 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:25.306872    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" podUID="f1421745-fcd8-41fc-ba64-ff1336e43684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 04:05:26.306043 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:26.306008    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 04:05:26.306480 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:26.306075    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" podUID="f1421745-fcd8-41fc-ba64-ff1336e43684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 04:05:27.306892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:27.306851    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 04:05:27.307398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:27.306919    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" podUID="f1421745-fcd8-41fc-ba64-ff1336e43684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 04:05:28.305713 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:28.305678    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 04:05:28.305923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:28.305733    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" podUID="f1421745-fcd8-41fc-ba64-ff1336e43684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 04:05:28.723585 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:05:28.723548    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:05:29.305688 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:29.305653    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 04:05:29.305887 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:29.305728    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" podUID="f1421745-fcd8-41fc-ba64-ff1336e43684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 04:05:30.305540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:30.305495    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Nov 01 04:05:30.306055 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:30.305581    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" podUID="f1421745-fcd8-41fc-ba64-ff1336e43684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Nov 01 04:05:31.310440 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:31.310395    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:31.310867 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:31.310744    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:31.311562 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:31.311542    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:05:37.752269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:37.752235    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"]
Nov 01 04:05:37.756770 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:37.756749    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:37.809911 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:37.809871    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"]
Nov 01 04:05:37.850126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:37.850079    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:05:37.850345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:37.850198    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:05:37.940206 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:37.940163    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-907dc852-mx9r4\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:37.940392 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:37.940338    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-907dc852-mx9r4\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:37.940502 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:37.940485    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhldr\" (UniqueName: \"kubernetes.io/projected/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-kube-api-access-lhldr\") pod \"maas-api-revoke-sub-907dc852-mx9r4\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:38.041663 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:38.041574    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-907dc852-mx9r4\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:38.041829 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:38.041671    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhldr\" (UniqueName: \"kubernetes.io/projected/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-kube-api-access-lhldr\") pod \"maas-api-revoke-sub-907dc852-mx9r4\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:38.041829 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:38.041780    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-907dc852-mx9r4\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:38.042393 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:38.042371    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-907dc852-mx9r4\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:38.044160 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:38.044140    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-907dc852-mx9r4\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:38.100031 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:38.099998    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhldr\" (UniqueName: \"kubernetes.io/projected/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-kube-api-access-lhldr\") pod \"maas-api-revoke-sub-907dc852-mx9r4\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:38.371069 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:38.370982    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:38.750245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:38.750215    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"]
Nov 01 04:05:38.750851 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:05:38.750819    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3eaac9_81e1_433a_ac79_ef1bbc8c773d.slice/crio-366ac8b5d495112462d06a9a91c9019cb9dda0a8c0a1c180b14461ac59ec40f8 WatchSource:0}: Error finding container 366ac8b5d495112462d06a9a91c9019cb9dda0a8c0a1c180b14461ac59ec40f8: Status 404 returned error can't find the container with id 366ac8b5d495112462d06a9a91c9019cb9dda0a8c0a1c180b14461ac59ec40f8
Nov 01 04:05:39.609905 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:39.609861    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4" event={"ID":"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d","Type":"ContainerStarted","Data":"be44e9a97a0e01d2d1b560cb3593fa2b44ab5c37a4d37ad087e5baa358e7c14b"}
Nov 01 04:05:39.610317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:39.609933    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4" event={"ID":"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d","Type":"ContainerStarted","Data":"366ac8b5d495112462d06a9a91c9019cb9dda0a8c0a1c180b14461ac59ec40f8"}
Nov 01 04:05:39.743413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:39.743352    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4" podStartSLOduration=2.743336809 podStartE2EDuration="2.743336809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:05:39.739336181 +0000 UTC m=+1408.301369198" watchObservedRunningTime="2026-11-01 04:05:39.743336809 +0000 UTC m=+1408.305369826"
Nov 01 04:05:40.618487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:40.618450    4882 generic.go:356] "Generic (PLEG): container finished" podID="3e3eaac9-81e1-433a-ac79-ef1bbc8c773d" containerID="be44e9a97a0e01d2d1b560cb3593fa2b44ab5c37a4d37ad087e5baa358e7c14b" exitCode=0
Nov 01 04:05:40.618867 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:40.618525    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4" event={"ID":"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d","Type":"ContainerDied","Data":"be44e9a97a0e01d2d1b560cb3593fa2b44ab5c37a4d37ad087e5baa358e7c14b"}
Nov 01 04:05:41.818476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:41.818445    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:41.972047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:41.972008    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-openshift-ca\") pod \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") "
Nov 01 04:05:41.972255 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:41.972198    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-xks-ca\") pod \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") "
Nov 01 04:05:41.972335 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:41.972288    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-kube-api-access-lhldr\" (UniqueName: \"kubernetes.io/projected/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-kube-api-access-lhldr\") pod \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\" (UID: \"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d\") "
Nov 01 04:05:41.972472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:41.972449    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-openshift-ca" pod "3e3eaac9-81e1-433a-ac79-ef1bbc8c773d" (UID: "3e3eaac9-81e1-433a-ac79-ef1bbc8c773d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:05:41.972986 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:41.972961    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:05:41.974707 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:41.974687    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-xks-ca" pod "3e3eaac9-81e1-433a-ac79-ef1bbc8c773d" (UID: "3e3eaac9-81e1-433a-ac79-ef1bbc8c773d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:05:41.975051 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:41.975033    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-kube-api-access-lhldr" pod "3e3eaac9-81e1-433a-ac79-ef1bbc8c773d" (UID: "3e3eaac9-81e1-433a-ac79-ef1bbc8c773d"). InnerVolumeSpecName "kube-api-access-lhldr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:05:42.074362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:42.074316    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:05:42.074362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:42.074354    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhldr\" (UniqueName: \"kubernetes.io/projected/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d-kube-api-access-lhldr\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:05:42.637446 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:42.637410    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"
Nov 01 04:05:42.637688 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:42.637445    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4" event={"ID":"3e3eaac9-81e1-433a-ac79-ef1bbc8c773d","Type":"ContainerDied","Data":"366ac8b5d495112462d06a9a91c9019cb9dda0a8c0a1c180b14461ac59ec40f8"}
Nov 01 04:05:42.637688 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:42.637495    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="366ac8b5d495112462d06a9a91c9019cb9dda0a8c0a1c180b14461ac59ec40f8"
Nov 01 04:05:56.047840 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:56.047804    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"]
Nov 01 04:05:56.093668 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:56.093629    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81fc914-nqg85"]
Nov 01 04:05:57.602984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:57.602944    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7875a8-9210-4eda-8206-20ee8820c99a" path="/var/lib/kubelet/pods/9d7875a8-9210-4eda-8206-20ee8820c99a/volumes"
Nov 01 04:05:58.719194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.719160    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"]
Nov 01 04:05:58.719903 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.719886    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3e3eaac9-81e1-433a-ac79-ef1bbc8c773d" containerName="revoke-keys"
Nov 01 04:05:58.719946 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.719904    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3eaac9-81e1-433a-ac79-ef1bbc8c773d" containerName="revoke-keys"
Nov 01 04:05:58.720037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.720028    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3eaac9-81e1-433a-ac79-ef1bbc8c773d" containerName="revoke-keys"
Nov 01 04:05:58.723673 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.723651    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.725867 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:05:58.725832    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:05:58.761512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.757855    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-certs\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.761512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.757917    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwwr\" (UniqueName: \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-kube-api-access-fpwwr\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.761512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.757947    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istiod-ca-cert\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.761512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.758007    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-credential-socket\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.761512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.758048    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-token\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.761512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.758089    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-envoy\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.761512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.758151    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-data\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.761512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.758204    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-podinfo\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.761512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.758236    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-socket\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.785318 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.785277    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4adf4c-openshift-default-dockercfg-nxzb4\""
Nov 01 04:05:58.813119 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.813078    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"]
Nov 01 04:05:58.859396 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.859360    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-certs\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.859641 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.859409    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwwr\" (UniqueName: \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-kube-api-access-fpwwr\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.859641 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.859445    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istiod-ca-cert\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.859770 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.859648    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-credential-socket\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.859770 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.859712    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-token\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.859886 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.859767    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-envoy\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.859886 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.859851    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-data\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.859989 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.859918    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-certs\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.859989 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.859938    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-podinfo\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.859989 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.859979    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-socket\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.860148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.859992    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-credential-socket\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.860270 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.860249    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-data\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.860330 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.860276    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-socket\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.860330 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.860275    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istiod-ca-cert\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.862006 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.861979    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-envoy\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.862364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.862342    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-podinfo\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.916973 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.916934    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-token\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:58.917266 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:58.917241    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwwr\" (UniqueName: \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-kube-api-access-fpwwr\") pod \"e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") " pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:59.057819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:59.057711    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:05:59.572078 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:05:59.572045    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0056df_1fe9_42e7_8e0f_ea47f63a1243.slice/crio-7ee82a1569d1520265f82cbe2d12213c90ff1918e8e4020b25ae62e4386f7f69 WatchSource:0}: Error finding container 7ee82a1569d1520265f82cbe2d12213c90ff1918e8e4020b25ae62e4386f7f69: Status 404 returned error can't find the container with id 7ee82a1569d1520265f82cbe2d12213c90ff1918e8e4020b25ae62e4386f7f69
Nov 01 04:05:59.572825 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:59.572797    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"]
Nov 01 04:05:59.575224 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:59.575190    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:05:59.575339 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:59.575255    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:05:59.575339 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:59.575283    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:05:59.891712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:59.891625    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" event={"ID":"ad0056df-1fe9-42e7-8e0f-ea47f63a1243","Type":"ContainerStarted","Data":"44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a"}
Nov 01 04:05:59.891712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:05:59.891687    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" event={"ID":"ad0056df-1fe9-42e7-8e0f-ea47f63a1243","Type":"ContainerStarted","Data":"7ee82a1569d1520265f82cbe2d12213c90ff1918e8e4020b25ae62e4386f7f69"}
Nov 01 04:06:00.007198 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:00.007141    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" podStartSLOduration=2.007128338 podStartE2EDuration="2.007128338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:06:00.004054039 +0000 UTC m=+1428.566087057" watchObservedRunningTime="2026-11-01 04:06:00.007128338 +0000 UTC m=+1428.569161356"
Nov 01 04:06:00.058938 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:00.058900    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:06:00.064525 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:00.064496    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:06:00.900192 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:00.900154    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:06:00.901296 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:00.901275    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:06:10.047040 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:10.047001    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"]
Nov 01 04:06:10.093105 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:10.093064    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-973dabb3-pxxkh"]
Nov 01 04:06:11.602510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:11.602478    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525a464b-f703-48aa-913e-015a3ccda41b" path="/var/lib/kubelet/pods/525a464b-f703-48aa-913e-015a3ccda41b/volumes"
Nov 01 04:06:19.048644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:19.048601    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"]
Nov 01 04:06:19.093914 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:19.093880    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6109dfd-d7x5h"]
Nov 01 04:06:19.604070 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:19.604038    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a926b3-8d49-4068-b19b-72709e72aca2" path="/var/lib/kubelet/pods/f4a926b3-8d49-4068-b19b-72709e72aca2/volumes"
Nov 01 04:06:24.504379 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.504343    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"]
Nov 01 04:06:24.510317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.510294    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.559360 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.559326    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3344e8-openshift-default-dockercfg-5jmgg\""
Nov 01 04:06:24.611017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.610985    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"]
Nov 01 04:06:24.615127 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.615101    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-data\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.615269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.615134    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-credential-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.615325 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.615300    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istiod-ca-cert\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.615379 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.615344    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-podinfo\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.615531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.615511    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-token\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.615638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.615621    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.615680 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.615667    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg4wg\" (UniqueName: \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-kube-api-access-bg4wg\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.615787 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.615770    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-certs\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.615826 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.615799    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-envoy\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.717603 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.717564    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-certs\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.717603 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.717613    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-envoy\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.717891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.717676    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-data\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.717891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.717703    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-credential-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.717891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.717819    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istiod-ca-cert\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.717891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.717851    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-podinfo\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.718097 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.717976    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-token\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.718097 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.718013    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-certs\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.718097 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.718080    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.718239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.718104    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-data\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.718239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.718109    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg4wg\" (UniqueName: \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-kube-api-access-bg4wg\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.718239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.718207    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-credential-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.718563 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.718537    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.718705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.718646    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istiod-ca-cert\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.720156 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.720134    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-envoy\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.720497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.720480    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-podinfo\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.775604 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.775534    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-token\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.775825 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.775800    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg4wg\" (UniqueName: \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-kube-api-access-bg4wg\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:24.824547 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:24.824508    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:25.256235 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:06:25.256204    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b4b9a4_ea51_4dd3_93ef_dee4c9cd9c35.slice/crio-5745f081b0e9ea627b882e7fc660082b8f741ff93db815086d32eacc58b669ac WatchSource:0}: Error finding container 5745f081b0e9ea627b882e7fc660082b8f741ff93db815086d32eacc58b669ac: Status 404 returned error can't find the container with id 5745f081b0e9ea627b882e7fc660082b8f741ff93db815086d32eacc58b669ac
Nov 01 04:06:25.256923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:25.256898    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"]
Nov 01 04:06:25.259040 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:25.259002    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:06:25.259142 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:25.259084    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:06:25.259142 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:25.259130    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:06:26.097850 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:26.097808    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx" event={"ID":"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35","Type":"ContainerStarted","Data":"93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff"}
Nov 01 04:06:26.098253 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:26.097875    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx" event={"ID":"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35","Type":"ContainerStarted","Data":"5745f081b0e9ea627b882e7fc660082b8f741ff93db815086d32eacc58b669ac"}
Nov 01 04:06:26.257912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:26.257853    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx" podStartSLOduration=2.257833048 podStartE2EDuration="2.257833048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:06:26.257024634 +0000 UTC m=+1454.819057663" watchObservedRunningTime="2026-11-01 04:06:26.257833048 +0000 UTC m=+1454.819866131"
Nov 01 04:06:26.825597 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:26.825557    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:26.827376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:26.827353    4882 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 04:06:26.827475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:26.827401    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx" podUID="62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 04:06:27.825860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:27.825822    4882 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 04:06:27.826344 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:27.825886    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx" podUID="62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 04:06:28.726201 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:06:28.726164    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:06:28.825210 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:28.825169    4882 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 04:06:28.825393 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:28.825255    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx" podUID="62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 04:06:29.825017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:29.824982    4882 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 04:06:29.825441 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:29.825046    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx" podUID="62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 04:06:30.825237 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:30.825194    4882 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 04:06:30.825791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:30.825279    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx" podUID="62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 04:06:31.830027 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:31.829995    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:31.830545 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:31.830405    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:31.831148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:31.831128    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:06:35.050847 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:35.050806    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"]
Nov 01 04:06:35.096926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:35.096892    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58600c40-b54tr"]
Nov 01 04:06:35.604164 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:35.604131    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2127a957-f579-46ce-befe-2437011ddfe9" path="/var/lib/kubelet/pods/2127a957-f579-46ce-befe-2437011ddfe9/volumes"
Nov 01 04:06:36.052752 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:36.052708    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"]
Nov 01 04:06:36.105380 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:36.105214    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8611e76-ccmq7"]
Nov 01 04:06:37.602116 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:37.602080    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5193c36-42c8-46da-9d1c-ff27a0799492" path="/var/lib/kubelet/pods/e5193c36-42c8-46da-9d1c-ff27a0799492/volumes"
Nov 01 04:06:45.046836 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:45.046793    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"]
Nov 01 04:06:45.093301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:45.093263    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-284101ce-dqvvg"]
Nov 01 04:06:45.600989 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:45.600939    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eeb7e4f-1b49-41d6-aa6c-8387544bfc11" path="/var/lib/kubelet/pods/5eeb7e4f-1b49-41d6-aa6c-8387544bfc11/volumes"
Nov 01 04:06:52.045478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.045362    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"]
Nov 01 04:06:52.091800 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.091761    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc8c1d0-rmfkv"]
Nov 01 04:06:52.198354 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.198315    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"]
Nov 01 04:06:52.204045 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.204022    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:52.252361 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.252331    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:06:52.253046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.253022    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:06:52.256115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.256087    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"]
Nov 01 04:06:52.295376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.295317    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:52.295659 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.295387    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:52.295743 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.295716    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58wtq\" (UniqueName: \"kubernetes.io/projected/4a64ac5e-dbc0-47a5-94f5-0019c9860217-kube-api-access-58wtq\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:52.397313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.397279    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58wtq\" (UniqueName: \"kubernetes.io/projected/4a64ac5e-dbc0-47a5-94f5-0019c9860217-kube-api-access-58wtq\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:52.397539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.397342    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:52.397539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.397365    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:52.398110 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.398090    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:52.399922 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.399902    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:52.457795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.457714    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wtq\" (UniqueName: \"kubernetes.io/projected/4a64ac5e-dbc0-47a5-94f5-0019c9860217-kube-api-access-58wtq\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:52.520444 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.520387    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:52.907561 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:52.907525    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"]
Nov 01 04:06:52.908586 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:06:52.908556    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a64ac5e_dbc0_47a5_94f5_0019c9860217.slice/crio-465d113d3afe6627546bf9bfacc2b1bdae2902db6a13b7e7812c88416fb97050 WatchSource:0}: Error finding container 465d113d3afe6627546bf9bfacc2b1bdae2902db6a13b7e7812c88416fb97050: Status 404 returned error can't find the container with id 465d113d3afe6627546bf9bfacc2b1bdae2902db6a13b7e7812c88416fb97050
Nov 01 04:06:53.355608 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.355561    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq" event={"ID":"4a64ac5e-dbc0-47a5-94f5-0019c9860217","Type":"ContainerStarted","Data":"0b06e6a29ee88e0cb50bf78407c35dd6a10b6232061f262d3b7650f14d06a9f3"}
Nov 01 04:06:53.356025 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.355664    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq" event={"ID":"4a64ac5e-dbc0-47a5-94f5-0019c9860217","Type":"ContainerStarted","Data":"465d113d3afe6627546bf9bfacc2b1bdae2902db6a13b7e7812c88416fb97050"}
Nov 01 04:06:53.468099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.468042    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq" podStartSLOduration=1.468025392 podStartE2EDuration="1.468025392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:06:53.463474685 +0000 UTC m=+1482.025507696" watchObservedRunningTime="2026-11-01 04:06:53.468025392 +0000 UTC m=+1482.030058454"
Nov 01 04:06:53.602299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.602263    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db616db-f37c-4b49-84c6-4f3684d6a295" path="/var/lib/kubelet/pods/6db616db-f37c-4b49-84c6-4f3684d6a295/volumes"
Nov 01 04:06:53.726748 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.726671    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"]
Nov 01 04:06:53.731518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.731494    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:53.785015 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.784983    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"]
Nov 01 04:06:53.818913 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.818872    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-182e51-chkc7\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:53.819111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.819011    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbk7\" (UniqueName: \"kubernetes.io/projected/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-kube-api-access-hkbk7\") pod \"maas-api-revoke-keys-e2e-api-a-182e51-chkc7\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:53.819261 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.819227    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-182e51-chkc7\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:53.920774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.920736    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-182e51-chkc7\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:53.920966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.920845    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbk7\" (UniqueName: \"kubernetes.io/projected/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-kube-api-access-hkbk7\") pod \"maas-api-revoke-keys-e2e-api-a-182e51-chkc7\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:53.921034 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.921017    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-182e51-chkc7\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:53.921705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.921677    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-182e51-chkc7\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:53.923184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.923165    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-182e51-chkc7\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:53.978085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:53.978009    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbk7\" (UniqueName: \"kubernetes.io/projected/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-kube-api-access-hkbk7\") pod \"maas-api-revoke-keys-e2e-api-a-182e51-chkc7\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:54.046965 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:54.046927    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:54.366067 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:54.365980    4882 generic.go:356] "Generic (PLEG): container finished" podID="4a64ac5e-dbc0-47a5-94f5-0019c9860217" containerID="0b06e6a29ee88e0cb50bf78407c35dd6a10b6232061f262d3b7650f14d06a9f3" exitCode=0
Nov 01 04:06:54.366460 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:54.366062    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq" event={"ID":"4a64ac5e-dbc0-47a5-94f5-0019c9860217","Type":"ContainerDied","Data":"0b06e6a29ee88e0cb50bf78407c35dd6a10b6232061f262d3b7650f14d06a9f3"}
Nov 01 04:06:54.423998 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:06:54.423946    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c363dd_4fbf_4180_8e61_3b2135f7ac17.slice/crio-add8c4dbd2d641342fce56b6ab06d84d11577fe284736da3f064e45ade295642 WatchSource:0}: Error finding container add8c4dbd2d641342fce56b6ab06d84d11577fe284736da3f064e45ade295642: Status 404 returned error can't find the container with id add8c4dbd2d641342fce56b6ab06d84d11577fe284736da3f064e45ade295642
Nov 01 04:06:54.465249 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:54.465217    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"]
Nov 01 04:06:55.377487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.377408    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7" event={"ID":"b9c363dd-4fbf-4180-8e61-3b2135f7ac17","Type":"ContainerStarted","Data":"0366e78c286312e272e9ec3b9b05f02737fb7f54bea198852bd7f6694ae97759"}
Nov 01 04:06:55.377946 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.377509    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7" event={"ID":"b9c363dd-4fbf-4180-8e61-3b2135f7ac17","Type":"ContainerStarted","Data":"add8c4dbd2d641342fce56b6ab06d84d11577fe284736da3f064e45ade295642"}
Nov 01 04:06:55.559803 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.559770    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:55.655768 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.655678    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a64ac5e-dbc0-47a5-94f5-0019c9860217-kube-api-access-58wtq\" (UniqueName: \"kubernetes.io/projected/4a64ac5e-dbc0-47a5-94f5-0019c9860217-kube-api-access-58wtq\") pod \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") "
Nov 01 04:06:55.655768 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.655730    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-openshift-ca\") pod \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") "
Nov 01 04:06:55.655962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.655894    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-xks-ca\") pod \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\" (UID: \"4a64ac5e-dbc0-47a5-94f5-0019c9860217\") "
Nov 01 04:06:55.656155 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.656120    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-openshift-ca" pod "4a64ac5e-dbc0-47a5-94f5-0019c9860217" (UID: "4a64ac5e-dbc0-47a5-94f5-0019c9860217"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:06:55.656801 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.656702    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:06:55.657972 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.657949    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a64ac5e-dbc0-47a5-94f5-0019c9860217-kube-api-access-58wtq" pod "4a64ac5e-dbc0-47a5-94f5-0019c9860217" (UID: "4a64ac5e-dbc0-47a5-94f5-0019c9860217"). InnerVolumeSpecName "kube-api-access-58wtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:06:55.658050 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.657990    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-xks-ca" pod "4a64ac5e-dbc0-47a5-94f5-0019c9860217" (UID: "4a64ac5e-dbc0-47a5-94f5-0019c9860217"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:06:55.757925 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.757891    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a64ac5e-dbc0-47a5-94f5-0019c9860217-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:06:55.757925 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:55.757919    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58wtq\" (UniqueName: \"kubernetes.io/projected/4a64ac5e-dbc0-47a5-94f5-0019c9860217-kube-api-access-58wtq\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:06:56.387568 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.387529    4882 generic.go:356] "Generic (PLEG): container finished" podID="b9c363dd-4fbf-4180-8e61-3b2135f7ac17" containerID="0366e78c286312e272e9ec3b9b05f02737fb7f54bea198852bd7f6694ae97759" exitCode=0
Nov 01 04:06:56.388075 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.387621    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7" event={"ID":"b9c363dd-4fbf-4180-8e61-3b2135f7ac17","Type":"ContainerDied","Data":"0366e78c286312e272e9ec3b9b05f02737fb7f54bea198852bd7f6694ae97759"}
Nov 01 04:06:56.389326 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.389303    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"
Nov 01 04:06:56.389489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.389334    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq" event={"ID":"4a64ac5e-dbc0-47a5-94f5-0019c9860217","Type":"ContainerDied","Data":"465d113d3afe6627546bf9bfacc2b1bdae2902db6a13b7e7812c88416fb97050"}
Nov 01 04:06:56.389489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.389376    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="465d113d3afe6627546bf9bfacc2b1bdae2902db6a13b7e7812c88416fb97050"
Nov 01 04:06:56.555981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.555951    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:56.673168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.673086    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-xks-ca\") pod \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") "
Nov 01 04:06:56.673320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.673189    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-kube-api-access-hkbk7\" (UniqueName: \"kubernetes.io/projected/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-kube-api-access-hkbk7\") pod \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") "
Nov 01 04:06:56.673320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.673262    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-openshift-ca\") pod \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\" (UID: \"b9c363dd-4fbf-4180-8e61-3b2135f7ac17\") "
Nov 01 04:06:56.673721 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.673691    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-openshift-ca" pod "b9c363dd-4fbf-4180-8e61-3b2135f7ac17" (UID: "b9c363dd-4fbf-4180-8e61-3b2135f7ac17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:06:56.674036 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.674013    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:06:56.675413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.675392    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-xks-ca" pod "b9c363dd-4fbf-4180-8e61-3b2135f7ac17" (UID: "b9c363dd-4fbf-4180-8e61-3b2135f7ac17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:06:56.676038 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.676018    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-kube-api-access-hkbk7" pod "b9c363dd-4fbf-4180-8e61-3b2135f7ac17" (UID: "b9c363dd-4fbf-4180-8e61-3b2135f7ac17"). InnerVolumeSpecName "kube-api-access-hkbk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:06:56.775725 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.775687    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:06:56.775725 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:56.775721    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkbk7\" (UniqueName: \"kubernetes.io/projected/b9c363dd-4fbf-4180-8e61-3b2135f7ac17-kube-api-access-hkbk7\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:06:57.399534 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:57.399500    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7" event={"ID":"b9c363dd-4fbf-4180-8e61-3b2135f7ac17","Type":"ContainerDied","Data":"add8c4dbd2d641342fce56b6ab06d84d11577fe284736da3f064e45ade295642"}
Nov 01 04:06:57.399534 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:57.399516    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"
Nov 01 04:06:57.400107 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:06:57.399589    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="add8c4dbd2d641342fce56b6ab06d84d11577fe284736da3f064e45ade295642"
Nov 01 04:06:58.724797 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:06:58.724759    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:07:02.049672 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:02.049634    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"]
Nov 01 04:07:02.095593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:02.095556    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659955d3-svhjq"]
Nov 01 04:07:03.046110 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:03.046070    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"]
Nov 01 04:07:03.092939 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:03.092903    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99f4129e-cqqcm"]
Nov 01 04:07:03.604057 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:03.604020    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbae6694-2406-4fa7-9794-8fea2fb69fe0" path="/var/lib/kubelet/pods/dbae6694-2406-4fa7-9794-8fea2fb69fe0/volumes"
Nov 01 04:07:03.604838 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:03.604817    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4f66b2-482e-4ad7-b0d4-f497d440e3ab" path="/var/lib/kubelet/pods/ec4f66b2-482e-4ad7-b0d4-f497d440e3ab/volumes"
Nov 01 04:07:12.327607 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:12.327570    4882 scope.go:122] "RemoveContainer" containerID="e27b7dcba89b40db11096c128fd74f3447c91093f58de138875a7de0f7b2705c"
Nov 01 04:07:12.338114 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:12.338094    4882 scope.go:122] "RemoveContainer" containerID="d206380fe68f546ba67635fa5e7ba2dd6a80d9737f3374a7f80cdb24ab87120a"
Nov 01 04:07:12.892146 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:12.892118    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:07:12.892363 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:12.892152    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:07:14.889664 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:14.889618    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"]
Nov 01 04:07:14.890468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:14.890422    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4a64ac5e-dbc0-47a5-94f5-0019c9860217" containerName="revoke-keys"
Nov 01 04:07:14.890468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:14.890466    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a64ac5e-dbc0-47a5-94f5-0019c9860217" containerName="revoke-keys"
Nov 01 04:07:14.890687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:14.890526    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b9c363dd-4fbf-4180-8e61-3b2135f7ac17" containerName="revoke-keys"
Nov 01 04:07:14.890687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:14.890535    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c363dd-4fbf-4180-8e61-3b2135f7ac17" containerName="revoke-keys"
Nov 01 04:07:14.890796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:14.890731    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a64ac5e-dbc0-47a5-94f5-0019c9860217" containerName="revoke-keys"
Nov 01 04:07:14.890796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:14.890757    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c363dd-4fbf-4180-8e61-3b2135f7ac17" containerName="revoke-keys"
Nov 01 04:07:14.896264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:14.896233    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:14.947550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:14.947516    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"]
Nov 01 04:07:14.987724 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:14.987675    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:07:14.987724 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:14.987688    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:07:15.014172 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:15.014133    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkwf\" (UniqueName: \"kubernetes.io/projected/b8958cd1-dbed-4f7a-8098-94a80bec26a5-kube-api-access-5nkwf\") pod \"maas-api-revoke-sub-c0d26b1d-9vmt4\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:15.014345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:15.014193    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0d26b1d-9vmt4\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:15.014423 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:15.014366    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0d26b1d-9vmt4\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:15.115748 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:15.115705    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkwf\" (UniqueName: \"kubernetes.io/projected/b8958cd1-dbed-4f7a-8098-94a80bec26a5-kube-api-access-5nkwf\") pod \"maas-api-revoke-sub-c0d26b1d-9vmt4\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:15.115919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:15.115767    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0d26b1d-9vmt4\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:15.115919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:15.115856    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0d26b1d-9vmt4\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:15.117111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:15.117081    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0d26b1d-9vmt4\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:15.118299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:15.118274    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0d26b1d-9vmt4\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:15.173563 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:15.173483    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkwf\" (UniqueName: \"kubernetes.io/projected/b8958cd1-dbed-4f7a-8098-94a80bec26a5-kube-api-access-5nkwf\") pod \"maas-api-revoke-sub-c0d26b1d-9vmt4\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:15.215226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:15.215181    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:15.591532 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:07:15.591491    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8958cd1_dbed_4f7a_8098_94a80bec26a5.slice/crio-6bd911aa1780a79554022883f6b1022159966c2f30193a8f03052d44f2af5d26 WatchSource:0}: Error finding container 6bd911aa1780a79554022883f6b1022159966c2f30193a8f03052d44f2af5d26: Status 404 returned error can't find the container with id 6bd911aa1780a79554022883f6b1022159966c2f30193a8f03052d44f2af5d26
Nov 01 04:07:15.591849 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:15.591822    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"]
Nov 01 04:07:16.563558 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:16.563524    4882 generic.go:356] "Generic (PLEG): container finished" podID="b8958cd1-dbed-4f7a-8098-94a80bec26a5" containerID="bfc40de66c245ff12fdb98e3901059a3796403c55f32850527296cc49e1ab9e5" exitCode=0
Nov 01 04:07:16.563979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:16.563579    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4" event={"ID":"b8958cd1-dbed-4f7a-8098-94a80bec26a5","Type":"ContainerDied","Data":"bfc40de66c245ff12fdb98e3901059a3796403c55f32850527296cc49e1ab9e5"}
Nov 01 04:07:16.563979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:16.563618    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4" event={"ID":"b8958cd1-dbed-4f7a-8098-94a80bec26a5","Type":"ContainerStarted","Data":"6bd911aa1780a79554022883f6b1022159966c2f30193a8f03052d44f2af5d26"}
Nov 01 04:07:17.764918 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:17.764885    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:17.858068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:17.857973    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-openshift-ca\") pod \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") "
Nov 01 04:07:17.858068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:17.858024    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-xks-ca\") pod \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") "
Nov 01 04:07:17.858068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:17.858072    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8958cd1-dbed-4f7a-8098-94a80bec26a5-kube-api-access-5nkwf\" (UniqueName: \"kubernetes.io/projected/b8958cd1-dbed-4f7a-8098-94a80bec26a5-kube-api-access-5nkwf\") pod \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\" (UID: \"b8958cd1-dbed-4f7a-8098-94a80bec26a5\") "
Nov 01 04:07:17.858456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:17.858408    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-openshift-ca" pod "b8958cd1-dbed-4f7a-8098-94a80bec26a5" (UID: "b8958cd1-dbed-4f7a-8098-94a80bec26a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:07:17.860392 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:17.860368    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8958cd1-dbed-4f7a-8098-94a80bec26a5-kube-api-access-5nkwf" pod "b8958cd1-dbed-4f7a-8098-94a80bec26a5" (UID: "b8958cd1-dbed-4f7a-8098-94a80bec26a5"). InnerVolumeSpecName "kube-api-access-5nkwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:07:17.860392 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:17.860374    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-xks-ca" pod "b8958cd1-dbed-4f7a-8098-94a80bec26a5" (UID: "b8958cd1-dbed-4f7a-8098-94a80bec26a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:07:17.960201 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:17.960160    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nkwf\" (UniqueName: \"kubernetes.io/projected/b8958cd1-dbed-4f7a-8098-94a80bec26a5-kube-api-access-5nkwf\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:07:17.960201 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:17.960200    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:07:17.960418 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:17.960216    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8958cd1-dbed-4f7a-8098-94a80bec26a5-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:07:18.582583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:18.582539    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"
Nov 01 04:07:18.582583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:18.582568    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4" event={"ID":"b8958cd1-dbed-4f7a-8098-94a80bec26a5","Type":"ContainerDied","Data":"6bd911aa1780a79554022883f6b1022159966c2f30193a8f03052d44f2af5d26"}
Nov 01 04:07:18.582848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:18.582662    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd911aa1780a79554022883f6b1022159966c2f30193a8f03052d44f2af5d26"
Nov 01 04:07:20.047146 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:20.047096    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"]
Nov 01 04:07:20.093077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:20.093028    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406d372b-t6mkb"]
Nov 01 04:07:21.046320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:21.046277    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"]
Nov 01 04:07:21.092518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:21.092483    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b1595d-qwtvz"]
Nov 01 04:07:21.600655 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:21.600612    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9e2a38-7d00-45ab-b84b-bf691400a3e9" path="/var/lib/kubelet/pods/9c9e2a38-7d00-45ab-b84b-bf691400a3e9/volumes"
Nov 01 04:07:21.601238 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:21.601215    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df6ffac2-1fff-4e03-b9eb-1ba04654256a" path="/var/lib/kubelet/pods/df6ffac2-1fff-4e03-b9eb-1ba04654256a/volumes"
Nov 01 04:07:28.724679 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:07:28.724634    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:07:31.047952 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:31.047913    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"]
Nov 01 04:07:31.097085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:31.097047    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a486244-rssgb"]
Nov 01 04:07:31.602822 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:31.602788    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57baa698-3b7c-4c44-9670-2863f988a88d" path="/var/lib/kubelet/pods/57baa698-3b7c-4c44-9670-2863f988a88d/volumes"
Nov 01 04:07:33.047080 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:33.047039    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"]
Nov 01 04:07:33.093265 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:33.093232    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de59a4cf-tphpx"]
Nov 01 04:07:33.602523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:33.602486    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0defc6-7634-4c8d-8b8a-0f74cb2a3410" path="/var/lib/kubelet/pods/ac0defc6-7634-4c8d-8b8a-0f74cb2a3410/volumes"
Nov 01 04:07:40.806014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:40.805978    4882 generic.go:356] "Generic (PLEG): container finished" podID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerID="44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a" exitCode=0
Nov 01 04:07:40.806532 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:40.806052    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" event={"ID":"ad0056df-1fe9-42e7-8e0f-ea47f63a1243","Type":"ContainerDied","Data":"44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a"}
Nov 01 04:07:40.806601 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:40.806585    4882 scope.go:122] "RemoveContainer" containerID="44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a"
Nov 01 04:07:40.808296 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:40.808270    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:07:40.808383 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:40.808345    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:07:40.808424 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:40.808382    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:07:41.826598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:41.826565    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" event={"ID":"ad0056df-1fe9-42e7-8e0f-ea47f63a1243","Type":"ContainerStarted","Data":"64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc"}
Nov 01 04:07:42.025450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.024678    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"]
Nov 01 04:07:42.029450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.026040    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b8958cd1-dbed-4f7a-8098-94a80bec26a5" containerName="revoke-keys"
Nov 01 04:07:42.029450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.026134    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8958cd1-dbed-4f7a-8098-94a80bec26a5" containerName="revoke-keys"
Nov 01 04:07:42.029450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.026556    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8958cd1-dbed-4f7a-8098-94a80bec26a5" containerName="revoke-keys"
Nov 01 04:07:42.073790 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.073745    4882 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 04:07:42.083922 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.083821    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 04:07:42.119560 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.104928    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:07:42.151611 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.131128    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.251959 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.251926    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"]
Nov 01 04:07:42.287312 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.281226    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-envoy\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.317594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.315788    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65wff\" (UniqueName: \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-kube-api-access-65wff\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.317594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.315926    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-socket\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.317594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.316045    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bfa667-16e5-4748-a44e-fa057ace864e-istio-podinfo\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.317594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.316101    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bfa667-16e5-4748-a44e-fa057ace864e-istiod-ca-cert\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.317594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.316332    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-data\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.317594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.316411    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-istio-token\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.317594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.316748    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-credential-socket\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.317594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.316855    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-certs\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.329242 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.329204    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dc980b-openshift-default-dockercfg-mtzk7\""
Nov 01 04:07:42.418852 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.418758    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-credential-socket\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.418852 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.418819    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-certs\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419048 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.418859    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-envoy\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419048 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.418900    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65wff\" (UniqueName: \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-kube-api-access-65wff\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419048 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.418945    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-socket\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419048 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.419042    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bfa667-16e5-4748-a44e-fa057ace864e-istio-podinfo\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.419077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bfa667-16e5-4748-a44e-fa057ace864e-istiod-ca-cert\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.419204    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-credential-socket\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.419214    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-data\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.419255    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-certs\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419618 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.419274    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-istio-token\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419618 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.419599    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-data\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419742 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.419624    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-socket\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.419901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.419879    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bfa667-16e5-4748-a44e-fa057ace864e-istiod-ca-cert\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.421314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.421291    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-envoy\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.422170 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.422146    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bfa667-16e5-4748-a44e-fa057ace864e-istio-podinfo\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.488911 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.488872    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-istio-token\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.489071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.488884    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65wff\" (UniqueName: \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-kube-api-access-65wff\") pod \"e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") " pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:42.633570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:42.633533    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:43.059679 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:43.059578    4882 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 04:07:43.059679 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:43.059655    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 04:07:43.089841 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:43.089808    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"]
Nov 01 04:07:43.091911 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:07:43.091885    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bfa667_16e5_4748_a44e_fa057ace864e.slice/crio-f78d8aa5203d75462a8013fbaeab5d7d4313225add2de6b9108a1047d28b8964 WatchSource:0}: Error finding container f78d8aa5203d75462a8013fbaeab5d7d4313225add2de6b9108a1047d28b8964: Status 404 returned error can't find the container with id f78d8aa5203d75462a8013fbaeab5d7d4313225add2de6b9108a1047d28b8964
Nov 01 04:07:43.094983 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:43.094948    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:07:43.095060 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:43.095018    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:07:43.095060 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:43.095043    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:07:43.843008 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:43.842967    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" event={"ID":"48bfa667-16e5-4748-a44e-fa057ace864e","Type":"ContainerStarted","Data":"5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba"}
Nov 01 04:07:43.843216 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:43.843028    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" event={"ID":"48bfa667-16e5-4748-a44e-fa057ace864e","Type":"ContainerStarted","Data":"f78d8aa5203d75462a8013fbaeab5d7d4313225add2de6b9108a1047d28b8964"}
Nov 01 04:07:43.959727 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:43.959624    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" podStartSLOduration=2.959609196 podStartE2EDuration="2.959609196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:07:43.955759512 +0000 UTC m=+1532.517792530" watchObservedRunningTime="2026-11-01 04:07:43.959609196 +0000 UTC m=+1532.521642213"
Nov 01 04:07:44.049926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:44.049891    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"]
Nov 01 04:07:44.054258 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:44.054228    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36db9bea-fbwlb"]
Nov 01 04:07:44.059316 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:44.059293    4882 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 04:07:44.059420 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:44.059340    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 04:07:44.634277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:44.634236    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:44.640120 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:44.640087    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:44.851287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:44.851254    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:44.852908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:44.852881    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:07:45.059554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:45.059519    4882 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 04:07:45.059746 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:45.059576    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 04:07:45.602043 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:45.602010    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4536851-8c8a-4dab-9766-9fa9d1ad157a" path="/var/lib/kubelet/pods/a4536851-8c8a-4dab-9766-9fa9d1ad157a/volumes"
Nov 01 04:07:45.900879 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:45.900793    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:07:46.058619 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:46.058580    4882 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 04:07:46.058798 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:46.058636    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 04:07:47.114194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:47.114150    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:07:47.118123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:47.118093    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:07:49.045046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:49.045007    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"]
Nov 01 04:07:49.091133 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:49.091096    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931cb99d-wbkx4"]
Nov 01 04:07:49.603980 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:49.603947    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc8e0d7-3eb5-4a9c-a098-c283ca327e22" path="/var/lib/kubelet/pods/dcc8e0d7-3eb5-4a9c-a098-c283ca327e22/volumes"
Nov 01 04:07:56.324803 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.324759    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"]
Nov 01 04:07:56.338942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.338705    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:07:56.384950 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.384894    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"]
Nov 01 04:07:56.424322 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.424037    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:07:56.424322 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.424263    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:07:56.486941 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.486904    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:07:56.487144 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.487004    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d46zp\" (UniqueName: \"kubernetes.io/projected/4805e0cb-7874-4741-a266-cc554fcc3391-kube-api-access-d46zp\") pod \"maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:07:56.487229 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.487211    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:07:56.589248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.589142    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:07:56.589489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.589285    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:07:56.589489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.589347    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d46zp\" (UniqueName: \"kubernetes.io/projected/4805e0cb-7874-4741-a266-cc554fcc3391-kube-api-access-d46zp\") pod \"maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:07:56.589981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.589953    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:07:56.592025 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.591999    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:07:56.646890 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.646847    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d46zp\" (UniqueName: \"kubernetes.io/projected/4805e0cb-7874-4741-a266-cc554fcc3391-kube-api-access-d46zp\") pod \"maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:07:56.654762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:56.654727    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:07:57.119381 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:57.119343    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"]
Nov 01 04:07:57.120227 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:07:57.120196    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4805e0cb_7874_4741_a266_cc554fcc3391.slice/crio-2f7cb4f9cb2e2779623bbe31b9a998caae3a2b80ec6056af0e1a3868200cd265 WatchSource:0}: Error finding container 2f7cb4f9cb2e2779623bbe31b9a998caae3a2b80ec6056af0e1a3868200cd265: Status 404 returned error can't find the container with id 2f7cb4f9cb2e2779623bbe31b9a998caae3a2b80ec6056af0e1a3868200cd265
Nov 01 04:07:58.031169 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:58.031131    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt" event={"ID":"4805e0cb-7874-4741-a266-cc554fcc3391","Type":"ContainerStarted","Data":"5a551b3b5d48dc0478b99c75f36907b0e2289d1066f524391eab0d70e8b6e5d5"}
Nov 01 04:07:58.031623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:58.031188    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt" event={"ID":"4805e0cb-7874-4741-a266-cc554fcc3391","Type":"ContainerStarted","Data":"2f7cb4f9cb2e2779623bbe31b9a998caae3a2b80ec6056af0e1a3868200cd265"}
Nov 01 04:07:58.048249 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:58.048216    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"]
Nov 01 04:07:58.094322 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:58.094283    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2087ee9d-5v2jh"]
Nov 01 04:07:58.150009 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:58.149943    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt" podStartSLOduration=2.149927861 podStartE2EDuration="2.149927861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:07:58.148303737 +0000 UTC m=+1546.710336757" watchObservedRunningTime="2026-11-01 04:07:58.149927861 +0000 UTC m=+1546.711960880"
Nov 01 04:07:58.725277 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:07:58.725244    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:07:59.041632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:59.041540    4882 generic.go:356] "Generic (PLEG): container finished" podID="4805e0cb-7874-4741-a266-cc554fcc3391" containerID="5a551b3b5d48dc0478b99c75f36907b0e2289d1066f524391eab0d70e8b6e5d5" exitCode=0
Nov 01 04:07:59.041632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:59.041615    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt" event={"ID":"4805e0cb-7874-4741-a266-cc554fcc3391","Type":"ContainerDied","Data":"5a551b3b5d48dc0478b99c75f36907b0e2289d1066f524391eab0d70e8b6e5d5"}
Nov 01 04:07:59.047669 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:59.047640    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"]
Nov 01 04:07:59.094010 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:59.093973    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e79d56ee-zglcz"]
Nov 01 04:07:59.602611 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:59.602573    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df4f391-88d7-410c-be67-aaa521a42547" path="/var/lib/kubelet/pods/1df4f391-88d7-410c-be67-aaa521a42547/volumes"
Nov 01 04:07:59.603224 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:07:59.603206    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98fd2689-9333-40a0-9045-2c5e7a191a98" path="/var/lib/kubelet/pods/98fd2689-9333-40a0-9045-2c5e7a191a98/volumes"
Nov 01 04:08:00.219801 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:00.219775    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:08:00.282802 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:00.282769    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4805e0cb-7874-4741-a266-cc554fcc3391-kube-api-access-d46zp\" (UniqueName: \"kubernetes.io/projected/4805e0cb-7874-4741-a266-cc554fcc3391-kube-api-access-d46zp\") pod \"4805e0cb-7874-4741-a266-cc554fcc3391\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") "
Nov 01 04:08:00.283005 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:00.282909    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-openshift-ca\") pod \"4805e0cb-7874-4741-a266-cc554fcc3391\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") "
Nov 01 04:08:00.283005 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:00.282957    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-xks-ca\") pod \"4805e0cb-7874-4741-a266-cc554fcc3391\" (UID: \"4805e0cb-7874-4741-a266-cc554fcc3391\") "
Nov 01 04:08:00.283333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:00.283307    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-openshift-ca" pod "4805e0cb-7874-4741-a266-cc554fcc3391" (UID: "4805e0cb-7874-4741-a266-cc554fcc3391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:00.283675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:00.283654    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:00.284970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:00.284942    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4805e0cb-7874-4741-a266-cc554fcc3391-kube-api-access-d46zp" pod "4805e0cb-7874-4741-a266-cc554fcc3391" (UID: "4805e0cb-7874-4741-a266-cc554fcc3391"). InnerVolumeSpecName "kube-api-access-d46zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:00.285072 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:00.285010    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-xks-ca" pod "4805e0cb-7874-4741-a266-cc554fcc3391" (UID: "4805e0cb-7874-4741-a266-cc554fcc3391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:08:00.385033 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:00.384937    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d46zp\" (UniqueName: \"kubernetes.io/projected/4805e0cb-7874-4741-a266-cc554fcc3391-kube-api-access-d46zp\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:00.385033 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:00.384977    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4805e0cb-7874-4741-a266-cc554fcc3391-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:01.058854 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:01.058824    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"
Nov 01 04:08:01.059059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:01.058859    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt" event={"ID":"4805e0cb-7874-4741-a266-cc554fcc3391","Type":"ContainerDied","Data":"2f7cb4f9cb2e2779623bbe31b9a998caae3a2b80ec6056af0e1a3868200cd265"}
Nov 01 04:08:01.059059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:01.058903    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f7cb4f9cb2e2779623bbe31b9a998caae3a2b80ec6056af0e1a3868200cd265"
Nov 01 04:08:11.056520 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:11.056480    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"]
Nov 01 04:08:11.098786 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:11.098744    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb96eb5d-6zm9c"]
Nov 01 04:08:11.660590 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:11.660545    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be5fb81f-7fe6-49b6-aa63-39d0f161104f" path="/var/lib/kubelet/pods/be5fb81f-7fe6-49b6-aa63-39d0f161104f/volumes"
Nov 01 04:08:12.374056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:12.374025    4882 scope.go:122] "RemoveContainer" containerID="3014a7ff7d6c04abec1366d50b765defe23eac93387dfc869416aa1ef17dd8a8"
Nov 01 04:08:12.386040 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:12.386016    4882 scope.go:122] "RemoveContainer" containerID="45bd7dcf219981af41c29c971309b16994f6e6d0c1e6328d20686c2c36b0bde8"
Nov 01 04:08:12.396683 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:12.396653    4882 scope.go:122] "RemoveContainer" containerID="fddff4c83894f2e36df50e595ccb0a841a1cca48bdff6e24c403769a942b0bea"
Nov 01 04:08:12.406970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:12.406946    4882 scope.go:122] "RemoveContainer" containerID="3e64177c10fe575bceef13ebbc4f4d9c367fac557fff7cb29448c9d71a196f69"
Nov 01 04:08:12.417811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:12.417787    4882 scope.go:122] "RemoveContainer" containerID="d53aab3937a722fa4d4a9d9d0eae5fac2f5f52f30d2633c86b550e0ba399f6d0"
Nov 01 04:08:12.428233 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:12.428208    4882 scope.go:122] "RemoveContainer" containerID="260545963f8eebe16d8fb97c906424f5b45142b2573829097b34ee5249cbbb91"
Nov 01 04:08:12.438098 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:12.438078    4882 scope.go:122] "RemoveContainer" containerID="b114e3d02140546ca7fe750ed991a57598e6c0f8e6487b14aac50b4756c14648"
Nov 01 04:08:20.506266 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:20.506187    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"]
Nov 01 04:08:20.552157 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:20.552108    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-k4nxq"]
Nov 01 04:08:21.255207 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:21.255166    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"]
Nov 01 04:08:21.255466 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:21.255417    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerName="istio-proxy" containerID="cri-o://d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed" gracePeriod=30
Nov 01 04:08:21.601527 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:21.601418    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a64ac5e-dbc0-47a5-94f5-0019c9860217" path="/var/lib/kubelet/pods/4a64ac5e-dbc0-47a5-94f5-0019c9860217/volumes"
Nov 01 04:08:26.751574 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.751515    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:08:26.808028 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.807985    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-token\" (UniqueName: \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-token\") pod \"405f0d74-fe60-4658-98ec-e133c24cd0fb\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") "
Nov 01 04:08:26.808253 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808115    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405f0d74-fe60-4658-98ec-e133c24cd0fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f0d74-fe60-4658-98ec-e133c24cd0fb-istiod-ca-cert\") pod \"405f0d74-fe60-4658-98ec-e133c24cd0fb\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") "
Nov 01 04:08:26.808253 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808141    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-envoy\") pod \"405f0d74-fe60-4658-98ec-e133c24cd0fb\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") "
Nov 01 04:08:26.808253 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808163    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-socket\") pod \"405f0d74-fe60-4658-98ec-e133c24cd0fb\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") "
Nov 01 04:08:26.808253 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808211    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-certs\") pod \"405f0d74-fe60-4658-98ec-e133c24cd0fb\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") "
Nov 01 04:08:26.808573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808255    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-credential-socket\") pod \"405f0d74-fe60-4658-98ec-e133c24cd0fb\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") "
Nov 01 04:08:26.808573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808344    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-podinfo\") pod \"405f0d74-fe60-4658-98ec-e133c24cd0fb\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") "
Nov 01 04:08:26.808573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808415    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-kube-api-access-qhpr2\" (UniqueName: \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-kube-api-access-qhpr2\") pod \"405f0d74-fe60-4658-98ec-e133c24cd0fb\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") "
Nov 01 04:08:26.808573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808489    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-socket" pod "405f0d74-fe60-4658-98ec-e133c24cd0fb" (UID: "405f0d74-fe60-4658-98ec-e133c24cd0fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:26.808573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808553    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405f0d74-fe60-4658-98ec-e133c24cd0fb-istiod-ca-cert" pod "405f0d74-fe60-4658-98ec-e133c24cd0fb" (UID: "405f0d74-fe60-4658-98ec-e133c24cd0fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:26.808573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808559    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-data\") pod \"405f0d74-fe60-4658-98ec-e133c24cd0fb\" (UID: \"405f0d74-fe60-4658-98ec-e133c24cd0fb\") "
Nov 01 04:08:26.808918 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808608    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-certs" pod "405f0d74-fe60-4658-98ec-e133c24cd0fb" (UID: "405f0d74-fe60-4658-98ec-e133c24cd0fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:26.808918 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808650    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-credential-socket" pod "405f0d74-fe60-4658-98ec-e133c24cd0fb" (UID: "405f0d74-fe60-4658-98ec-e133c24cd0fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:26.809035 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.808946    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-data" pod "405f0d74-fe60-4658-98ec-e133c24cd0fb" (UID: "405f0d74-fe60-4658-98ec-e133c24cd0fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:26.809514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.809486    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:26.809651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.809517    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:26.809651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.809536    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f0d74-fe60-4658-98ec-e133c24cd0fb-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:26.809651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.809560    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:26.809651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.809574    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:26.811684 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.811641    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-envoy" pod "405f0d74-fe60-4658-98ec-e133c24cd0fb" (UID: "405f0d74-fe60-4658-98ec-e133c24cd0fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:26.812059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.812031    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-kube-api-access-qhpr2" pod "405f0d74-fe60-4658-98ec-e133c24cd0fb" (UID: "405f0d74-fe60-4658-98ec-e133c24cd0fb"). InnerVolumeSpecName "kube-api-access-qhpr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:26.812515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.812492    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-token" pod "405f0d74-fe60-4658-98ec-e133c24cd0fb" (UID: "405f0d74-fe60-4658-98ec-e133c24cd0fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:26.813366 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.813341    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-podinfo" pod "405f0d74-fe60-4658-98ec-e133c24cd0fb" (UID: "405f0d74-fe60-4658-98ec-e133c24cd0fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:08:26.890617 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.890570    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"]
Nov 01 04:08:26.897414 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.897374    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-182e51-chkc7"]
Nov 01 04:08:26.911394 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.911356    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:26.911544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.911400    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:26.911544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.911418    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhpr2\" (UniqueName: \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-kube-api-access-qhpr2\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:26.911544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:26.911452    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f0d74-fe60-4658-98ec-e133c24cd0fb-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:27.344690 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:27.344645    4882 generic.go:356] "Generic (PLEG): container finished" podID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerID="d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed" exitCode=0
Nov 01 04:08:27.344885 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:27.344703    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"
Nov 01 04:08:27.344885 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:27.344703    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" event={"ID":"405f0d74-fe60-4658-98ec-e133c24cd0fb","Type":"ContainerDied","Data":"d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed"}
Nov 01 04:08:27.344885 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:27.344777    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q" event={"ID":"405f0d74-fe60-4658-98ec-e133c24cd0fb","Type":"ContainerDied","Data":"4772a692431e0222655069d0b43e2a3975c8805dfd9a4698432ddfcf5c64ba6a"}
Nov 01 04:08:27.344885 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:27.344811    4882 scope.go:122] "RemoveContainer" containerID="d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed"
Nov 01 04:08:27.359073 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:27.359051    4882 scope.go:122] "RemoveContainer" containerID="d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed"
Nov 01 04:08:27.359412 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:08:27.359387    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed\": container with ID starting with d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed not found: ID does not exist" containerID="d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed"
Nov 01 04:08:27.359563 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:27.359474    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed"} err="failed to get container status \"d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed\": rpc error: code = NotFound desc = could not find container \"d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed\": container with ID starting with d16ce933a30edd68c311105344cb2227c5368cf6d1703d0e663f07347d725fed not found: ID does not exist"
Nov 01 04:08:27.503081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:27.503047    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"]
Nov 01 04:08:27.548631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:27.548594    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-gvz2q"]
Nov 01 04:08:27.601973 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:27.601879    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" path="/var/lib/kubelet/pods/405f0d74-fe60-4658-98ec-e133c24cd0fb/volumes"
Nov 01 04:08:27.602592 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:27.602570    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c363dd-4fbf-4180-8e61-3b2135f7ac17" path="/var/lib/kubelet/pods/b9c363dd-4fbf-4180-8e61-3b2135f7ac17/volumes"
Nov 01 04:08:28.725029 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:08:28.724986    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:08:29.050331 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:29.050227    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"]
Nov 01 04:08:29.093178 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:29.093134    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d98b25e-m4dpm"]
Nov 01 04:08:29.603053 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:29.603021    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="120956bd-4b22-467c-a6c5-56ea377c9c7d" path="/var/lib/kubelet/pods/120956bd-4b22-467c-a6c5-56ea377c9c7d/volumes"
Nov 01 04:08:33.713670 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:33.713628    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"]
Nov 01 04:08:33.714207 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:33.713877    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy" containerID="cri-o://873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5" gracePeriod=30
Nov 01 04:08:34.979026 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:34.978983    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"]
Nov 01 04:08:34.980014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:34.979982    4882 cpu_manager.go:414] "containerMap: removing container" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerName="istio-proxy"
Nov 01 04:08:34.980014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:34.980012    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerName="istio-proxy"
Nov 01 04:08:34.980231 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:34.980103    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4805e0cb-7874-4741-a266-cc554fcc3391" containerName="revoke-keys"
Nov 01 04:08:34.980231 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:34.980113    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4805e0cb-7874-4741-a266-cc554fcc3391" containerName="revoke-keys"
Nov 01 04:08:34.980341 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:34.980285    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4805e0cb-7874-4741-a266-cc554fcc3391" containerName="revoke-keys"
Nov 01 04:08:34.980341 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:34.980301    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="405f0d74-fe60-4658-98ec-e133c24cd0fb" containerName="istio-proxy"
Nov 01 04:08:35.010599 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.006741    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:35.078474 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.071269    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:08:35.092452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.087902    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"]
Nov 01 04:08:35.092902 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.092875    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:08:35.174415 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.156497    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxksj\" (UniqueName: \"kubernetes.io/projected/03c4eea3-e198-46d1-adb5-f74a2d8ed561-kube-api-access-hxksj\") pod \"maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:35.175062 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.175032    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:35.175243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.175226    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:35.278254 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.278163    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:35.279126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.279099    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:35.280271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.280251    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxksj\" (UniqueName: \"kubernetes.io/projected/03c4eea3-e198-46d1-adb5-f74a2d8ed561-kube-api-access-hxksj\") pod \"maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:35.280781 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.279995    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:35.282053 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.282016    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:35.337664 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.337628    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxksj\" (UniqueName: \"kubernetes.io/projected/03c4eea3-e198-46d1-adb5-f74a2d8ed561-kube-api-access-hxksj\") pod \"maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:35.355604 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.355567    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:35.736855 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.736819    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"]
Nov 01 04:08:35.738736 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:08:35.738695    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c4eea3_e198_46d1_adb5_f74a2d8ed561.slice/crio-2fe1a7907bd46404cb8657e52ea404ff2910fb80895e55ec7faa93b73919412a WatchSource:0}: Error finding container 2fe1a7907bd46404cb8657e52ea404ff2910fb80895e55ec7faa93b73919412a: Status 404 returned error can't find the container with id 2fe1a7907bd46404cb8657e52ea404ff2910fb80895e55ec7faa93b73919412a
Nov 01 04:08:35.837385 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.837346    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"]
Nov 01 04:08:35.837684 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:35.837574    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" podUID="6c799659-ba33-4ce6-826c-58897edd2134" containerName="istio-proxy" containerID="cri-o://0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27" gracePeriod=30
Nov 01 04:08:36.229263 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:36.229227    4882 patch_prober.go:29] interesting pod/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:08:36.229713 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:36.229282    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:08:36.418402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:36.418294    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q" event={"ID":"03c4eea3-e198-46d1-adb5-f74a2d8ed561","Type":"ContainerStarted","Data":"1ef46474d2af463e5ed69269b63511f655fbefa9dba7ba3755b7add36699b6df"}
Nov 01 04:08:36.418402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:36.418393    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q" event={"ID":"03c4eea3-e198-46d1-adb5-f74a2d8ed561","Type":"ContainerStarted","Data":"2fe1a7907bd46404cb8657e52ea404ff2910fb80895e55ec7faa93b73919412a"}
Nov 01 04:08:36.527799 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:36.527743    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q" podStartSLOduration=2.527729151 podStartE2EDuration="2.527729151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:08:36.526352276 +0000 UTC m=+1585.088385293" watchObservedRunningTime="2026-11-01 04:08:36.527729151 +0000 UTC m=+1585.089762167"
Nov 01 04:08:37.427716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:37.427672    4882 generic.go:356] "Generic (PLEG): container finished" podID="03c4eea3-e198-46d1-adb5-f74a2d8ed561" containerID="1ef46474d2af463e5ed69269b63511f655fbefa9dba7ba3755b7add36699b6df" exitCode=0
Nov 01 04:08:37.428095 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:37.427747    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q" event={"ID":"03c4eea3-e198-46d1-adb5-f74a2d8ed561","Type":"ContainerDied","Data":"1ef46474d2af463e5ed69269b63511f655fbefa9dba7ba3755b7add36699b6df"}
Nov 01 04:08:38.602062 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:38.602034    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:38.736280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:38.736246    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-openshift-ca\") pod \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") "
Nov 01 04:08:38.736507 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:38.736480    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-xks-ca\") pod \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") "
Nov 01 04:08:38.736631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:38.736604    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c4eea3-e198-46d1-adb5-f74a2d8ed561-kube-api-access-hxksj\" (UniqueName: \"kubernetes.io/projected/03c4eea3-e198-46d1-adb5-f74a2d8ed561-kube-api-access-hxksj\") pod \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\" (UID: \"03c4eea3-e198-46d1-adb5-f74a2d8ed561\") "
Nov 01 04:08:38.737346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:38.737086    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-openshift-ca" pod "03c4eea3-e198-46d1-adb5-f74a2d8ed561" (UID: "03c4eea3-e198-46d1-adb5-f74a2d8ed561"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:38.737838 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:38.737793    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:38.739341 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:38.739309    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-xks-ca" pod "03c4eea3-e198-46d1-adb5-f74a2d8ed561" (UID: "03c4eea3-e198-46d1-adb5-f74a2d8ed561"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:08:38.739756 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:38.739732    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c4eea3-e198-46d1-adb5-f74a2d8ed561-kube-api-access-hxksj" pod "03c4eea3-e198-46d1-adb5-f74a2d8ed561" (UID: "03c4eea3-e198-46d1-adb5-f74a2d8ed561"). InnerVolumeSpecName "kube-api-access-hxksj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:38.839417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:38.839391    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c4eea3-e198-46d1-adb5-f74a2d8ed561-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:38.839417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:38.839417    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxksj\" (UniqueName: \"kubernetes.io/projected/03c4eea3-e198-46d1-adb5-f74a2d8ed561-kube-api-access-hxksj\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:38.996440 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:38.996397    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:08:39.143993 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.143953    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-kube-api-access-7khnb\" (UniqueName: \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-kube-api-access-7khnb\") pod \"4f68cfc4-6033-4820-bb68-e19695d59202\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") "
Nov 01 04:08:39.144171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144006    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-credential-socket\") pod \"4f68cfc4-6033-4820-bb68-e19695d59202\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") "
Nov 01 04:08:39.144171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144118    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-certs\") pod \"4f68cfc4-6033-4820-bb68-e19695d59202\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") "
Nov 01 04:08:39.144306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144277    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f68cfc4-6033-4820-bb68-e19695d59202-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f68cfc4-6033-4820-bb68-e19695d59202-istio-podinfo\") pod \"4f68cfc4-6033-4820-bb68-e19695d59202\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") "
Nov 01 04:08:39.144378 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144358    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f68cfc4-6033-4820-bb68-e19695d59202-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f68cfc4-6033-4820-bb68-e19695d59202-istiod-ca-cert\") pod \"4f68cfc4-6033-4820-bb68-e19695d59202\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") "
Nov 01 04:08:39.144464 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144384    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-credential-socket" pod "4f68cfc4-6033-4820-bb68-e19695d59202" (UID: "4f68cfc4-6033-4820-bb68-e19695d59202"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:39.144464 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144405    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-istio-token\" (UniqueName: \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-istio-token\") pod \"4f68cfc4-6033-4820-bb68-e19695d59202\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") "
Nov 01 04:08:39.144464 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144411    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-certs" pod "4f68cfc4-6033-4820-bb68-e19695d59202" (UID: "4f68cfc4-6033-4820-bb68-e19695d59202"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:39.144627 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144478    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-envoy\") pod \"4f68cfc4-6033-4820-bb68-e19695d59202\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") "
Nov 01 04:08:39.144683 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144635    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-socket\") pod \"4f68cfc4-6033-4820-bb68-e19695d59202\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") "
Nov 01 04:08:39.144745 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144680    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-data\") pod \"4f68cfc4-6033-4820-bb68-e19695d59202\" (UID: \"4f68cfc4-6033-4820-bb68-e19695d59202\") "
Nov 01 04:08:39.144802 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144767    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f68cfc4-6033-4820-bb68-e19695d59202-istiod-ca-cert" pod "4f68cfc4-6033-4820-bb68-e19695d59202" (UID: "4f68cfc4-6033-4820-bb68-e19695d59202"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:39.145065 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.144947    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-socket" pod "4f68cfc4-6033-4820-bb68-e19695d59202" (UID: "4f68cfc4-6033-4820-bb68-e19695d59202"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:39.145178 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.145148    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-data" pod "4f68cfc4-6033-4820-bb68-e19695d59202" (UID: "4f68cfc4-6033-4820-bb68-e19695d59202"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:39.145519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.145498    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f68cfc4-6033-4820-bb68-e19695d59202-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:39.145622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.145524    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:39.145622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.145539    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:39.145622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.145552    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:39.145622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.145565    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:39.146859 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.146829    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-kube-api-access-7khnb" pod "4f68cfc4-6033-4820-bb68-e19695d59202" (UID: "4f68cfc4-6033-4820-bb68-e19695d59202"). InnerVolumeSpecName "kube-api-access-7khnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:39.147006 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.146987    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-envoy" pod "4f68cfc4-6033-4820-bb68-e19695d59202" (UID: "4f68cfc4-6033-4820-bb68-e19695d59202"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:39.147093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.147077    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f68cfc4-6033-4820-bb68-e19695d59202-istio-podinfo" pod "4f68cfc4-6033-4820-bb68-e19695d59202" (UID: "4f68cfc4-6033-4820-bb68-e19695d59202"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:08:39.147488 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.147463    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-istio-token" pod "4f68cfc4-6033-4820-bb68-e19695d59202" (UID: "4f68cfc4-6033-4820-bb68-e19695d59202"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:39.246923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.246836    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:39.246923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.246869    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f68cfc4-6033-4820-bb68-e19695d59202-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:39.246923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.246879    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7khnb\" (UniqueName: \"kubernetes.io/projected/4f68cfc4-6033-4820-bb68-e19695d59202-kube-api-access-7khnb\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:39.246923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.246888    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f68cfc4-6033-4820-bb68-e19695d59202-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:39.443976 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.443937    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"
Nov 01 04:08:39.444168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.443942    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q" event={"ID":"03c4eea3-e198-46d1-adb5-f74a2d8ed561","Type":"ContainerDied","Data":"2fe1a7907bd46404cb8657e52ea404ff2910fb80895e55ec7faa93b73919412a"}
Nov 01 04:08:39.444168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.444080    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fe1a7907bd46404cb8657e52ea404ff2910fb80895e55ec7faa93b73919412a"
Nov 01 04:08:39.445972 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.445941    4882 generic.go:356] "Generic (PLEG): container finished" podID="4f68cfc4-6033-4820-bb68-e19695d59202" containerID="873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5" exitCode=0
Nov 01 04:08:39.446093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.445978    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"
Nov 01 04:08:39.446093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.446016    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" event={"ID":"4f68cfc4-6033-4820-bb68-e19695d59202","Type":"ContainerDied","Data":"873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5"}
Nov 01 04:08:39.446093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.446069    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c" event={"ID":"4f68cfc4-6033-4820-bb68-e19695d59202","Type":"ContainerDied","Data":"201d2272e6085c033978d464841b5e84eae79337ffa6de058502c06ccc6ad6f1"}
Nov 01 04:08:39.446189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.446097    4882 scope.go:122] "RemoveContainer" containerID="873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5"
Nov 01 04:08:39.458746 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.458728    4882 scope.go:122] "RemoveContainer" containerID="9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191"
Nov 01 04:08:39.469313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.469288    4882 scope.go:122] "RemoveContainer" containerID="873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5"
Nov 01 04:08:39.469670 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:08:39.469647    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5\": container with ID starting with 873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5 not found: ID does not exist" containerID="873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5"
Nov 01 04:08:39.469770 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.469681    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5"} err="failed to get container status \"873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5\": rpc error: code = NotFound desc = could not find container \"873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5\": container with ID starting with 873a2b57192978aef37120157713124d2ae85e5ab03d4c71d1efa7b289fe44d5 not found: ID does not exist"
Nov 01 04:08:39.469770 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.469709    4882 scope.go:122] "RemoveContainer" containerID="9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191"
Nov 01 04:08:39.470012 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:08:39.469993    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191\": container with ID starting with 9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191 not found: ID does not exist" containerID="9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191"
Nov 01 04:08:39.470067 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.470015    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191"} err="failed to get container status \"9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191\": rpc error: code = NotFound desc = could not find container \"9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191\": container with ID starting with 9b3b1f5e8d3d6e34f6154435c770c84a2fd70ce5bd0593b191b447124adac191 not found: ID does not exist"
Nov 01 04:08:39.650554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.650511    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"]
Nov 01 04:08:39.654881 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:39.654851    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-182e51-openshift-default-64c5d5457c-w4k9c"]
Nov 01 04:08:41.125952 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.125883    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:08:41.172723 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.172686    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-certs\") pod \"6c799659-ba33-4ce6-826c-58897edd2134\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") "
Nov 01 04:08:41.172940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.172772    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-socket\") pod \"6c799659-ba33-4ce6-826c-58897edd2134\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") "
Nov 01 04:08:41.172940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.172835    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-kube-api-access-v9j6p\" (UniqueName: \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-kube-api-access-v9j6p\") pod \"6c799659-ba33-4ce6-826c-58897edd2134\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") "
Nov 01 04:08:41.172940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.172856    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-envoy\") pod \"6c799659-ba33-4ce6-826c-58897edd2134\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") "
Nov 01 04:08:41.173180 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.172995    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c799659-ba33-4ce6-826c-58897edd2134-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c799659-ba33-4ce6-826c-58897edd2134-istiod-ca-cert\") pod \"6c799659-ba33-4ce6-826c-58897edd2134\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") "
Nov 01 04:08:41.173180 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.173025    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-certs" pod "6c799659-ba33-4ce6-826c-58897edd2134" (UID: "6c799659-ba33-4ce6-826c-58897edd2134"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:41.173180 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.173125    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-socket" pod "6c799659-ba33-4ce6-826c-58897edd2134" (UID: "6c799659-ba33-4ce6-826c-58897edd2134"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:41.173180 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.173138    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-data\") pod \"6c799659-ba33-4ce6-826c-58897edd2134\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") "
Nov 01 04:08:41.173396 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.173220    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-istio-token\" (UniqueName: \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-istio-token\") pod \"6c799659-ba33-4ce6-826c-58897edd2134\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") "
Nov 01 04:08:41.173396 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.173313    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c799659-ba33-4ce6-826c-58897edd2134-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c799659-ba33-4ce6-826c-58897edd2134-istio-podinfo\") pod \"6c799659-ba33-4ce6-826c-58897edd2134\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") "
Nov 01 04:08:41.173396 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.173333    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-data" pod "6c799659-ba33-4ce6-826c-58897edd2134" (UID: "6c799659-ba33-4ce6-826c-58897edd2134"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:41.173596 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.173463    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-credential-socket\") pod \"6c799659-ba33-4ce6-826c-58897edd2134\" (UID: \"6c799659-ba33-4ce6-826c-58897edd2134\") "
Nov 01 04:08:41.173596 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.173473    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c799659-ba33-4ce6-826c-58897edd2134-istiod-ca-cert" pod "6c799659-ba33-4ce6-826c-58897edd2134" (UID: "6c799659-ba33-4ce6-826c-58897edd2134"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:41.173708 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.173681    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-credential-socket" pod "6c799659-ba33-4ce6-826c-58897edd2134" (UID: "6c799659-ba33-4ce6-826c-58897edd2134"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:41.174199 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.174174    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:41.174307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.174204    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:41.174307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.174222    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:41.174307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.174237    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:41.174307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.174252    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c799659-ba33-4ce6-826c-58897edd2134-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:41.175405 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.175378    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-envoy" pod "6c799659-ba33-4ce6-826c-58897edd2134" (UID: "6c799659-ba33-4ce6-826c-58897edd2134"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:41.175554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.175492    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c799659-ba33-4ce6-826c-58897edd2134-istio-podinfo" pod "6c799659-ba33-4ce6-826c-58897edd2134" (UID: "6c799659-ba33-4ce6-826c-58897edd2134"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:08:41.175554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.175503    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-istio-token" pod "6c799659-ba33-4ce6-826c-58897edd2134" (UID: "6c799659-ba33-4ce6-826c-58897edd2134"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:41.175554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.175540    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-kube-api-access-v9j6p" pod "6c799659-ba33-4ce6-826c-58897edd2134" (UID: "6c799659-ba33-4ce6-826c-58897edd2134"). InnerVolumeSpecName "kube-api-access-v9j6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:41.275549 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.275513    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:41.275549 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.275543    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c799659-ba33-4ce6-826c-58897edd2134-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:41.275549 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.275553    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9j6p\" (UniqueName: \"kubernetes.io/projected/6c799659-ba33-4ce6-826c-58897edd2134-kube-api-access-v9j6p\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:41.275795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.275563    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c799659-ba33-4ce6-826c-58897edd2134-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:41.464620 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.464578    4882 generic.go:356] "Generic (PLEG): container finished" podID="6c799659-ba33-4ce6-826c-58897edd2134" containerID="0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27" exitCode=0
Nov 01 04:08:41.464797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.464626    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"
Nov 01 04:08:41.464797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.464661    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" event={"ID":"6c799659-ba33-4ce6-826c-58897edd2134","Type":"ContainerDied","Data":"0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27"}
Nov 01 04:08:41.464797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.464747    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r" event={"ID":"6c799659-ba33-4ce6-826c-58897edd2134","Type":"ContainerDied","Data":"595eb225af2fe9979273aa0699eed19d7b049c8d5140e91f358b75ef46ebfa10"}
Nov 01 04:08:41.464797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.464773    4882 scope.go:122] "RemoveContainer" containerID="0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27"
Nov 01 04:08:41.476984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.476965    4882 scope.go:122] "RemoveContainer" containerID="0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27"
Nov 01 04:08:41.477313 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:08:41.477288    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27\": container with ID starting with 0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27 not found: ID does not exist" containerID="0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27"
Nov 01 04:08:41.477377 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.477327    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27"} err="failed to get container status \"0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27\": rpc error: code = NotFound desc = could not find container \"0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27\": container with ID starting with 0980c1f5d84c5f818a58b79cf71d0a0b9a166803abcc6f8b5c003a47018dfb27 not found: ID does not exist"
Nov 01 04:08:41.601269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.601233    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" path="/var/lib/kubelet/pods/4f68cfc4-6033-4820-bb68-e19695d59202/volumes"
Nov 01 04:08:41.621879 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.621847    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"]
Nov 01 04:08:41.627823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:41.627798    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-htv6r"]
Nov 01 04:08:43.600644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:43.600602    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c799659-ba33-4ce6-826c-58897edd2134" path="/var/lib/kubelet/pods/6c799659-ba33-4ce6-826c-58897edd2134/volumes"
Nov 01 04:08:47.691992 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:47.691951    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"]
Nov 01 04:08:47.692602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:47.692256    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy" containerID="cri-o://90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d" gracePeriod=30
Nov 01 04:08:52.409119 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:52.409078    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"]
Nov 01 04:08:52.409602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:52.409323    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerName="istio-proxy" containerID="cri-o://4b0169f70fd676e471b6a85310c52f052c99e6cac6d286724238f630ee2a45cd" gracePeriod=30
Nov 01 04:08:52.442645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:52.442608    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:08:52.442825 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:52.442664    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:08:52.455173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:52.455134    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"]
Nov 01 04:08:52.455357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:52.455327    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" podUID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" containerName="istio-proxy" containerID="cri-o://dce86ae9a8d77a60fe019005018bf304f2af3e76b6406d692ebb9f2d066737f0" gracePeriod=30
Nov 01 04:08:52.458806 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:52.458775    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"]
Nov 01 04:08:52.459048 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:52.459013    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" podUID="551c4d53-6477-4592-99c4-486fd57ba643" containerName="istio-proxy" containerID="cri-o://ae3f108bd6c8da76564c970a948cf1511e73e74a80282f4f0cf44d57dd7bd295" gracePeriod=30
Nov 01 04:08:52.501497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:52.501461    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"]
Nov 01 04:08:52.501685 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:52.501665    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy" containerID="cri-o://d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089" gracePeriod=30
Nov 01 04:08:53.025152 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.025119    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:08:53.178926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.178891    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-socket\") pod \"948bb297-3490-4419-9cbd-5e11c367c538\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") "
Nov 01 04:08:53.179129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.179043    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/948bb297-3490-4419-9cbd-5e11c367c538-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948bb297-3490-4419-9cbd-5e11c367c538-istio-podinfo\") pod \"948bb297-3490-4419-9cbd-5e11c367c538\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") "
Nov 01 04:08:53.179129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.179082    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-certs\") pod \"948bb297-3490-4419-9cbd-5e11c367c538\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") "
Nov 01 04:08:53.179259 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.179135    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-credential-socket\") pod \"948bb297-3490-4419-9cbd-5e11c367c538\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") "
Nov 01 04:08:53.179259 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.179242    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/948bb297-3490-4419-9cbd-5e11c367c538-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948bb297-3490-4419-9cbd-5e11c367c538-istiod-ca-cert\") pod \"948bb297-3490-4419-9cbd-5e11c367c538\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") "
Nov 01 04:08:53.179375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.179318    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-data\") pod \"948bb297-3490-4419-9cbd-5e11c367c538\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") "
Nov 01 04:08:53.179375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.179361    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-kube-api-access-nn58w\" (UniqueName: \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-kube-api-access-nn58w\") pod \"948bb297-3490-4419-9cbd-5e11c367c538\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") "
Nov 01 04:08:53.179509 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.179485    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-envoy\") pod \"948bb297-3490-4419-9cbd-5e11c367c538\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") "
Nov 01 04:08:53.179566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.179525    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-istio-token\" (UniqueName: \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-istio-token\") pod \"948bb297-3490-4419-9cbd-5e11c367c538\" (UID: \"948bb297-3490-4419-9cbd-5e11c367c538\") "
Nov 01 04:08:53.179784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.179760    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-certs" pod "948bb297-3490-4419-9cbd-5e11c367c538" (UID: "948bb297-3490-4419-9cbd-5e11c367c538"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:53.180447 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.180185    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948bb297-3490-4419-9cbd-5e11c367c538-istiod-ca-cert" pod "948bb297-3490-4419-9cbd-5e11c367c538" (UID: "948bb297-3490-4419-9cbd-5e11c367c538"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:53.180447 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.180201    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:53.180780 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.180516    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-socket" pod "948bb297-3490-4419-9cbd-5e11c367c538" (UID: "948bb297-3490-4419-9cbd-5e11c367c538"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:53.180780 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.180592    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-data" pod "948bb297-3490-4419-9cbd-5e11c367c538" (UID: "948bb297-3490-4419-9cbd-5e11c367c538"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:53.181099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.181070    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-credential-socket" pod "948bb297-3490-4419-9cbd-5e11c367c538" (UID: "948bb297-3490-4419-9cbd-5e11c367c538"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:53.182622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.182597    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/948bb297-3490-4419-9cbd-5e11c367c538-istio-podinfo" pod "948bb297-3490-4419-9cbd-5e11c367c538" (UID: "948bb297-3490-4419-9cbd-5e11c367c538"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:08:53.183391 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.183362    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-istio-token" pod "948bb297-3490-4419-9cbd-5e11c367c538" (UID: "948bb297-3490-4419-9cbd-5e11c367c538"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:53.183523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.183404    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-kube-api-access-nn58w" pod "948bb297-3490-4419-9cbd-5e11c367c538" (UID: "948bb297-3490-4419-9cbd-5e11c367c538"). InnerVolumeSpecName "kube-api-access-nn58w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:53.183582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.183566    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-envoy" pod "948bb297-3490-4419-9cbd-5e11c367c538" (UID: "948bb297-3490-4419-9cbd-5e11c367c538"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:53.281762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.281670    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:53.281762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.281703    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948bb297-3490-4419-9cbd-5e11c367c538-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:53.281762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.281712    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:53.281762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.281721    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn58w\" (UniqueName: \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-kube-api-access-nn58w\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:53.281762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.281733    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:53.281762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.281741    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948bb297-3490-4419-9cbd-5e11c367c538-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:53.281762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.281750    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948bb297-3490-4419-9cbd-5e11c367c538-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:53.281762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.281760    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948bb297-3490-4419-9cbd-5e11c367c538-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:53.580974 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.580873    4882 generic.go:356] "Generic (PLEG): container finished" podID="948bb297-3490-4419-9cbd-5e11c367c538" containerID="90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d" exitCode=0
Nov 01 04:08:53.580974 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.580938    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"
Nov 01 04:08:53.580974 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.580960    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" event={"ID":"948bb297-3490-4419-9cbd-5e11c367c538","Type":"ContainerDied","Data":"90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d"}
Nov 01 04:08:53.581499 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.581043    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk" event={"ID":"948bb297-3490-4419-9cbd-5e11c367c538","Type":"ContainerDied","Data":"4d33351925a8e39bb9d1deeb24c8dd16ccfe62ff8c0aada0123b4c3dce3bbb42"}
Nov 01 04:08:53.581499 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.581067    4882 scope.go:122] "RemoveContainer" containerID="90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d"
Nov 01 04:08:53.592853 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.592834    4882 scope.go:122] "RemoveContainer" containerID="90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d"
Nov 01 04:08:53.593155 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:08:53.593127    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d\": container with ID starting with 90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d not found: ID does not exist" containerID="90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d"
Nov 01 04:08:53.593195 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.593169    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d"} err="failed to get container status \"90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d\": rpc error: code = NotFound desc = could not find container \"90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d\": container with ID starting with 90c1293075825601217eedd5c4f89ca2d0dfb2b784d1cdf1c61d3018e487029d not found: ID does not exist"
Nov 01 04:08:53.737004 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.736964    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"]
Nov 01 04:08:53.743096 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:53.743063    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-wwbpk"]
Nov 01 04:08:54.443504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:54.443461    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:08:54.443661 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:54.443538    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" podUID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:08:55.601769 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:55.601736    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948bb297-3490-4419-9cbd-5e11c367c538" path="/var/lib/kubelet/pods/948bb297-3490-4419-9cbd-5e11c367c538/volumes"
Nov 01 04:08:56.047827 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:56.047790    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"]
Nov 01 04:08:56.093949 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:56.093914    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705c0aa1-66rf2"]
Nov 01 04:08:56.134552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:56.134518    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:08:56.134731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:56.134580    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:08:57.094686 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.094647    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:08:57.095105 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.094712    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:08:57.500441 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.500388    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 04:08:57.500616 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.500460    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" podUID="551c4d53-6477-4592-99c4-486fd57ba643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 04:08:57.616106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.615761    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6be760-8e8c-46d0-a7d4-877e3660f9d0" path="/var/lib/kubelet/pods/bb6be760-8e8c-46d0-a7d4-877e3660f9d0/volumes"
Nov 01 04:08:57.626763 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.626725    4882 generic.go:356] "Generic (PLEG): container finished" podID="551c4d53-6477-4592-99c4-486fd57ba643" containerID="ae3f108bd6c8da76564c970a948cf1511e73e74a80282f4f0cf44d57dd7bd295" exitCode=0
Nov 01 04:08:57.626918 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.626807    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" event={"ID":"551c4d53-6477-4592-99c4-486fd57ba643","Type":"ContainerDied","Data":"ae3f108bd6c8da76564c970a948cf1511e73e74a80282f4f0cf44d57dd7bd295"}
Nov 01 04:08:57.628930 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.628908    4882 generic.go:356] "Generic (PLEG): container finished" podID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" containerID="dce86ae9a8d77a60fe019005018bf304f2af3e76b6406d692ebb9f2d066737f0" exitCode=0
Nov 01 04:08:57.629123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.628978    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" event={"ID":"a7a77365-4bae-4d90-95c9-43f6eb75d25a","Type":"ContainerDied","Data":"dce86ae9a8d77a60fe019005018bf304f2af3e76b6406d692ebb9f2d066737f0"}
Nov 01 04:08:57.631337 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.631317    4882 generic.go:356] "Generic (PLEG): container finished" podID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerID="4b0169f70fd676e471b6a85310c52f052c99e6cac6d286724238f630ee2a45cd" exitCode=0
Nov 01 04:08:57.631453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.631411    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" event={"ID":"d8fed9c9-bba5-4025-a338-8d7c7a831941","Type":"ContainerDied","Data":"4b0169f70fd676e471b6a85310c52f052c99e6cac6d286724238f630ee2a45cd"}
Nov 01 04:08:57.832711 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.832682    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:08:57.879865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.879836    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:08:57.934307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.934280    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 04:08:57.937938 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.937912    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 04:08:57.949504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949475    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-data\") pod \"551c4d53-6477-4592-99c4-486fd57ba643\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") "
Nov 01 04:08:57.949636 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949576    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-kube-api-access-g2qtg\" (UniqueName: \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-kube-api-access-g2qtg\") pod \"551c4d53-6477-4592-99c4-486fd57ba643\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") "
Nov 01 04:08:57.949636 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949598    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-kube-api-access-r8hvk\" (UniqueName: \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-kube-api-access-r8hvk\") pod \"d8fed9c9-bba5-4025-a338-8d7c7a831941\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") "
Nov 01 04:08:57.949636 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949616    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-socket\") pod \"d8fed9c9-bba5-4025-a338-8d7c7a831941\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") "
Nov 01 04:08:57.949804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949634    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-socket\") pod \"551c4d53-6477-4592-99c4-486fd57ba643\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") "
Nov 01 04:08:57.949804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949669    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-envoy\") pod \"d8fed9c9-bba5-4025-a338-8d7c7a831941\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") "
Nov 01 04:08:57.949804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949695    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8fed9c9-bba5-4025-a338-8d7c7a831941-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fed9c9-bba5-4025-a338-8d7c7a831941-istiod-ca-cert\") pod \"d8fed9c9-bba5-4025-a338-8d7c7a831941\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") "
Nov 01 04:08:57.949804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949731    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-credential-socket\") pod \"d8fed9c9-bba5-4025-a338-8d7c7a831941\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") "
Nov 01 04:08:57.949804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949747    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-data" pod "551c4d53-6477-4592-99c4-486fd57ba643" (UID: "551c4d53-6477-4592-99c4-486fd57ba643"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:57.949804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949769    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-token\" (UniqueName: \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-token\") pod \"d8fed9c9-bba5-4025-a338-8d7c7a831941\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") "
Nov 01 04:08:57.950171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949805    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-envoy\") pod \"551c4d53-6477-4592-99c4-486fd57ba643\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") "
Nov 01 04:08:57.950171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949846    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551c4d53-6477-4592-99c4-486fd57ba643-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551c4d53-6477-4592-99c4-486fd57ba643-istiod-ca-cert\") pod \"551c4d53-6477-4592-99c4-486fd57ba643\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") "
Nov 01 04:08:57.950171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949902    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-certs\") pod \"551c4d53-6477-4592-99c4-486fd57ba643\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") "
Nov 01 04:08:57.950171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949909    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-socket" pod "d8fed9c9-bba5-4025-a338-8d7c7a831941" (UID: "d8fed9c9-bba5-4025-a338-8d7c7a831941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:57.950171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949933    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-credential-socket\") pod \"551c4d53-6477-4592-99c4-486fd57ba643\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") "
Nov 01 04:08:57.950171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.949962    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-data\") pod \"d8fed9c9-bba5-4025-a338-8d7c7a831941\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") "
Nov 01 04:08:57.950171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.950053    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/551c4d53-6477-4592-99c4-486fd57ba643-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551c4d53-6477-4592-99c4-486fd57ba643-istio-podinfo\") pod \"551c4d53-6477-4592-99c4-486fd57ba643\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") "
Nov 01 04:08:57.950171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.950098    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-credential-socket" pod "d8fed9c9-bba5-4025-a338-8d7c7a831941" (UID: "d8fed9c9-bba5-4025-a338-8d7c7a831941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:57.950587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.950181    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-certs\") pod \"d8fed9c9-bba5-4025-a338-8d7c7a831941\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") "
Nov 01 04:08:57.950587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.950222    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-podinfo\") pod \"d8fed9c9-bba5-4025-a338-8d7c7a831941\" (UID: \"d8fed9c9-bba5-4025-a338-8d7c7a831941\") "
Nov 01 04:08:57.950587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.950282    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-istio-token\" (UniqueName: \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-istio-token\") pod \"551c4d53-6477-4592-99c4-486fd57ba643\" (UID: \"551c4d53-6477-4592-99c4-486fd57ba643\") "
Nov 01 04:08:57.950587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.950298    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-credential-socket" pod "551c4d53-6477-4592-99c4-486fd57ba643" (UID: "551c4d53-6477-4592-99c4-486fd57ba643"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:57.950587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.950326    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fed9c9-bba5-4025-a338-8d7c7a831941-istiod-ca-cert" pod "d8fed9c9-bba5-4025-a338-8d7c7a831941" (UID: "d8fed9c9-bba5-4025-a338-8d7c7a831941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:57.950836 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.950625    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-certs" pod "551c4d53-6477-4592-99c4-486fd57ba643" (UID: "551c4d53-6477-4592-99c4-486fd57ba643"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:57.950836 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.950667    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551c4d53-6477-4592-99c4-486fd57ba643-istiod-ca-cert" pod "551c4d53-6477-4592-99c4-486fd57ba643" (UID: "551c4d53-6477-4592-99c4-486fd57ba643"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:57.950940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.950860    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-data" pod "d8fed9c9-bba5-4025-a338-8d7c7a831941" (UID: "d8fed9c9-bba5-4025-a338-8d7c7a831941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:57.951048 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.951019    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-certs" pod "d8fed9c9-bba5-4025-a338-8d7c7a831941" (UID: "d8fed9c9-bba5-4025-a338-8d7c7a831941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:57.951540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.951294    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:57.951540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.951297    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-socket" pod "551c4d53-6477-4592-99c4-486fd57ba643" (UID: "551c4d53-6477-4592-99c4-486fd57ba643"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:57.951540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.951318    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:57.951540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.951335    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:57.951540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.951349    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fed9c9-bba5-4025-a338-8d7c7a831941-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:57.951540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.951363    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:57.951540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.951377    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551c4d53-6477-4592-99c4-486fd57ba643-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:57.951540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.951392    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:57.951540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.951406    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:57.951540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.951420    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:57.954173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.953975    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-envoy" pod "551c4d53-6477-4592-99c4-486fd57ba643" (UID: "551c4d53-6477-4592-99c4-486fd57ba643"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:57.954173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.954049    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/551c4d53-6477-4592-99c4-486fd57ba643-istio-podinfo" pod "551c4d53-6477-4592-99c4-486fd57ba643" (UID: "551c4d53-6477-4592-99c4-486fd57ba643"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:08:57.954173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.954102    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-envoy" pod "d8fed9c9-bba5-4025-a338-8d7c7a831941" (UID: "d8fed9c9-bba5-4025-a338-8d7c7a831941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:57.955044 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.955023    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-podinfo" pod "d8fed9c9-bba5-4025-a338-8d7c7a831941" (UID: "d8fed9c9-bba5-4025-a338-8d7c7a831941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:08:57.955233 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.955219    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-istio-token" pod "551c4d53-6477-4592-99c4-486fd57ba643" (UID: "551c4d53-6477-4592-99c4-486fd57ba643"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:57.955504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.955483    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-kube-api-access-r8hvk" pod "d8fed9c9-bba5-4025-a338-8d7c7a831941" (UID: "d8fed9c9-bba5-4025-a338-8d7c7a831941"). InnerVolumeSpecName "kube-api-access-r8hvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:57.955580 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.955562    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-token" pod "d8fed9c9-bba5-4025-a338-8d7c7a831941" (UID: "d8fed9c9-bba5-4025-a338-8d7c7a831941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:57.955906 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:57.955883    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-kube-api-access-g2qtg" pod "551c4d53-6477-4592-99c4-486fd57ba643" (UID: "551c4d53-6477-4592-99c4-486fd57ba643"). InnerVolumeSpecName "kube-api-access-g2qtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:58.052081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052046    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d36b171-563c-4230-b2fc-aa810dceaa0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d36b171-563c-4230-b2fc-aa810dceaa0b-istiod-ca-cert\") pod \"4d36b171-563c-4230-b2fc-aa810dceaa0b\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") "
Nov 01 04:08:58.052298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052092    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istiod-ca-cert\") pod \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") "
Nov 01 04:08:58.052298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052123    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-podinfo\") pod \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") "
Nov 01 04:08:58.052298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052141    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-data\") pod \"4d36b171-563c-4230-b2fc-aa810dceaa0b\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") "
Nov 01 04:08:58.052298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052199    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-credential-socket\") pod \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") "
Nov 01 04:08:58.052298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052269    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-credential-socket\") pod \"4d36b171-563c-4230-b2fc-aa810dceaa0b\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") "
Nov 01 04:08:58.052654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052333    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-token\" (UniqueName: \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-token\") pod \"4d36b171-563c-4230-b2fc-aa810dceaa0b\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") "
Nov 01 04:08:58.052654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052396    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-socket\") pod \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") "
Nov 01 04:08:58.052654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052458    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-data" pod "4d36b171-563c-4230-b2fc-aa810dceaa0b" (UID: "4d36b171-563c-4230-b2fc-aa810dceaa0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:58.052654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052478    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-credential-socket" pod "a7a77365-4bae-4d90-95c9-43f6eb75d25a" (UID: "a7a77365-4bae-4d90-95c9-43f6eb75d25a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:58.052654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052469    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-socket\") pod \"4d36b171-563c-4230-b2fc-aa810dceaa0b\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") "
Nov 01 04:08:58.052654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052531    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istiod-ca-cert" pod "a7a77365-4bae-4d90-95c9-43f6eb75d25a" (UID: "a7a77365-4bae-4d90-95c9-43f6eb75d25a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:58.052654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052547    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-kube-api-access-v6hq9\" (UniqueName: \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-kube-api-access-v6hq9\") pod \"4d36b171-563c-4230-b2fc-aa810dceaa0b\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") "
Nov 01 04:08:58.052654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052552    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-credential-socket" pod "4d36b171-563c-4230-b2fc-aa810dceaa0b" (UID: "4d36b171-563c-4230-b2fc-aa810dceaa0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:58.052654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052622    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-certs\") pod \"4d36b171-563c-4230-b2fc-aa810dceaa0b\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") "
Nov 01 04:08:58.053125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052668    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d36b171-563c-4230-b2fc-aa810dceaa0b-istiod-ca-cert" pod "4d36b171-563c-4230-b2fc-aa810dceaa0b" (UID: "4d36b171-563c-4230-b2fc-aa810dceaa0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:58.053125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052681    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-socket" pod "4d36b171-563c-4230-b2fc-aa810dceaa0b" (UID: "4d36b171-563c-4230-b2fc-aa810dceaa0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:58.053125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052676    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-podinfo\") pod \"4d36b171-563c-4230-b2fc-aa810dceaa0b\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") "
Nov 01 04:08:58.053125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052745    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-envoy\") pod \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") "
Nov 01 04:08:58.053125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052830    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-socket" pod "a7a77365-4bae-4d90-95c9-43f6eb75d25a" (UID: "a7a77365-4bae-4d90-95c9-43f6eb75d25a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:58.053125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052860    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-data\") pod \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") "
Nov 01 04:08:58.053125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052898    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-certs" pod "4d36b171-563c-4230-b2fc-aa810dceaa0b" (UID: "4d36b171-563c-4230-b2fc-aa810dceaa0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:58.053125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.052923    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-token\" (UniqueName: \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-token\") pod \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") "
Nov 01 04:08:58.053125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053014    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-certs\") pod \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") "
Nov 01 04:08:58.053125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053049    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-data" pod "a7a77365-4bae-4d90-95c9-43f6eb75d25a" (UID: "a7a77365-4bae-4d90-95c9-43f6eb75d25a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:58.053125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053123    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-envoy\") pod \"4d36b171-563c-4230-b2fc-aa810dceaa0b\" (UID: \"4d36b171-563c-4230-b2fc-aa810dceaa0b\") "
Nov 01 04:08:58.053741 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053179    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-kube-api-access-zgswm\" (UniqueName: \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-kube-api-access-zgswm\") pod \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\" (UID: \"a7a77365-4bae-4d90-95c9-43f6eb75d25a\") "
Nov 01 04:08:58.053741 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053343    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-certs" pod "a7a77365-4bae-4d90-95c9-43f6eb75d25a" (UID: "a7a77365-4bae-4d90-95c9-43f6eb75d25a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:58.053887 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053866    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.053954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053893    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.053954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053911    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2qtg\" (UniqueName: \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-kube-api-access-g2qtg\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.053954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053927    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8hvk\" (UniqueName: \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-kube-api-access-r8hvk\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.053954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053943    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053958    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053972    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053986    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551c4d53-6477-4592-99c4-486fd57ba643-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.053999    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.054013    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551c4d53-6477-4592-99c4-486fd57ba643-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.054027    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fed9c9-bba5-4025-a338-8d7c7a831941-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.054039    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d36b171-563c-4230-b2fc-aa810dceaa0b-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.054052    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.054066    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551c4d53-6477-4592-99c4-486fd57ba643-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.054079    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.054093    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.054106    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.054120    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.054908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.054133    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.055536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.055111    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-token" pod "4d36b171-563c-4230-b2fc-aa810dceaa0b" (UID: "4d36b171-563c-4230-b2fc-aa810dceaa0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:58.055536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.055163    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-podinfo" pod "4d36b171-563c-4230-b2fc-aa810dceaa0b" (UID: "4d36b171-563c-4230-b2fc-aa810dceaa0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:08:58.055536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.055262    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-podinfo" pod "a7a77365-4bae-4d90-95c9-43f6eb75d25a" (UID: "a7a77365-4bae-4d90-95c9-43f6eb75d25a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:08:58.055646 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.055532    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-envoy" pod "a7a77365-4bae-4d90-95c9-43f6eb75d25a" (UID: "a7a77365-4bae-4d90-95c9-43f6eb75d25a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:58.055685 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.055656    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-kube-api-access-v6hq9" pod "4d36b171-563c-4230-b2fc-aa810dceaa0b" (UID: "4d36b171-563c-4230-b2fc-aa810dceaa0b"). InnerVolumeSpecName "kube-api-access-v6hq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:58.055777 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.055760    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-envoy" pod "4d36b171-563c-4230-b2fc-aa810dceaa0b" (UID: "4d36b171-563c-4230-b2fc-aa810dceaa0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:58.056416 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.056398    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-token" pod "a7a77365-4bae-4d90-95c9-43f6eb75d25a" (UID: "a7a77365-4bae-4d90-95c9-43f6eb75d25a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:58.056750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.056732    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-kube-api-access-zgswm" pod "a7a77365-4bae-4d90-95c9-43f6eb75d25a" (UID: "a7a77365-4bae-4d90-95c9-43f6eb75d25a"). InnerVolumeSpecName "kube-api-access-zgswm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:58.155523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.155484    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.155523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.155518    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgswm\" (UniqueName: \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-kube-api-access-zgswm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.155523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.155529    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.156264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.155538    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.156264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.155547    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6hq9\" (UniqueName: \"kubernetes.io/projected/4d36b171-563c-4230-b2fc-aa810dceaa0b-kube-api-access-v6hq9\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.156264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.155558    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d36b171-563c-4230-b2fc-aa810dceaa0b-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.156264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.155566    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.156264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.155575    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a77365-4bae-4d90-95c9-43f6eb75d25a-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:08:58.641579 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.641543    4882 generic.go:356] "Generic (PLEG): container finished" podID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerID="d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089" exitCode=0
Nov 01 04:08:58.641769 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.641599    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"
Nov 01 04:08:58.641769 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.641631    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" event={"ID":"4d36b171-563c-4230-b2fc-aa810dceaa0b","Type":"ContainerDied","Data":"d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089"}
Nov 01 04:08:58.641769 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.641717    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh" event={"ID":"4d36b171-563c-4230-b2fc-aa810dceaa0b","Type":"ContainerDied","Data":"9642449ca731899256ddf5d9ad15be8426ea37853e819910337e9fbc53a91315"}
Nov 01 04:08:58.641769 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.641756    4882 scope.go:122] "RemoveContainer" containerID="d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089"
Nov 01 04:08:58.643575 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.643551    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"
Nov 01 04:08:58.643675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.643612    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v" event={"ID":"d8fed9c9-bba5-4025-a338-8d7c7a831941","Type":"ContainerDied","Data":"d441b36ebc34cbd5c322257226c108e01d82a7e15578991c42fde7a1e69b1d58"}
Nov 01 04:08:58.645286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.645258    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l" event={"ID":"551c4d53-6477-4592-99c4-486fd57ba643","Type":"ContainerDied","Data":"82459156427dfb23d6a9cd28d5fdd829153c4cc6bd2151a34a4b898336b4852d"}
Nov 01 04:08:58.645397 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.645298    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"
Nov 01 04:08:58.647706 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.647666    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"
Nov 01 04:08:58.647706 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.647672    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd" event={"ID":"a7a77365-4bae-4d90-95c9-43f6eb75d25a","Type":"ContainerDied","Data":"0bd84773eff1fc37f9d60c6f553f96e55a9112554c58a1112aaa8a487f30a970"}
Nov 01 04:08:58.657133 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.657111    4882 scope.go:122] "RemoveContainer" containerID="d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089"
Nov 01 04:08:58.657637 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:08:58.657557    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089\": container with ID starting with d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089 not found: ID does not exist" containerID="d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089"
Nov 01 04:08:58.657637 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.657596    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089"} err="failed to get container status \"d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089\": rpc error: code = NotFound desc = could not find container \"d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089\": container with ID starting with d525b9fae076f69047feb109c7091c82835dba41d6a7d3ae881f582975126089 not found: ID does not exist"
Nov 01 04:08:58.657637 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.657637    4882 scope.go:122] "RemoveContainer" containerID="4b0169f70fd676e471b6a85310c52f052c99e6cac6d286724238f630ee2a45cd"
Nov 01 04:08:58.818606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.818178    4882 scope.go:122] "RemoveContainer" containerID="ae3f108bd6c8da76564c970a948cf1511e73e74a80282f4f0cf44d57dd7bd295"
Nov 01 04:08:58.823125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.823053    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"]
Nov 01 04:08:58.824558 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:08:58.824519    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:08:58.831155 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.831128    4882 scope.go:122] "RemoveContainer" containerID="dce86ae9a8d77a60fe019005018bf304f2af3e76b6406d692ebb9f2d066737f0"
Nov 01 04:08:58.857479 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.857442    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-8c95l"]
Nov 01 04:08:58.959605 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.959569    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"]
Nov 01 04:08:58.967473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:58.967438    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-v5x4v"]
Nov 01 04:08:59.066942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:59.066842    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"]
Nov 01 04:08:59.072905 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:59.072870    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-f75fh"]
Nov 01 04:08:59.227166 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:59.227128    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"]
Nov 01 04:08:59.232877 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:59.232846    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-9r9fd"]
Nov 01 04:08:59.601847 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:59.601811    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" path="/var/lib/kubelet/pods/4d36b171-563c-4230-b2fc-aa810dceaa0b/volumes"
Nov 01 04:08:59.602353 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:59.602338    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551c4d53-6477-4592-99c4-486fd57ba643" path="/var/lib/kubelet/pods/551c4d53-6477-4592-99c4-486fd57ba643/volumes"
Nov 01 04:08:59.602810 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:59.602797    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" path="/var/lib/kubelet/pods/a7a77365-4bae-4d90-95c9-43f6eb75d25a/volumes"
Nov 01 04:08:59.603573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:08:59.603555    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" path="/var/lib/kubelet/pods/d8fed9c9-bba5-4025-a338-8d7c7a831941/volumes"
Nov 01 04:09:06.071163 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:06.071124    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"]
Nov 01 04:09:06.071641 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:06.071410    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy" containerID="cri-o://7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a" gracePeriod=30
Nov 01 04:09:09.766978 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:09.766935    4882 generic.go:356] "Generic (PLEG): container finished" podID="48bfa667-16e5-4748-a44e-fa057ace864e" containerID="5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba" exitCode=0
Nov 01 04:09:09.767457 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:09.767009    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" event={"ID":"48bfa667-16e5-4748-a44e-fa057ace864e","Type":"ContainerDied","Data":"5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba"}
Nov 01 04:09:09.767526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:09.767479    4882 scope.go:122] "RemoveContainer" containerID="5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba"
Nov 01 04:09:09.769152 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:09.769112    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:09:09.769280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:09.769200    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:09:09.769280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:09.769249    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:09:10.794645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:10.794603    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" event={"ID":"48bfa667-16e5-4748-a44e-fa057ace864e","Type":"ContainerStarted","Data":"89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc"}
Nov 01 04:09:11.116102 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.116064    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 01 04:09:11.116299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.116128    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 01 04:09:11.397056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.396980    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:09:11.500759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.500720    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-kube-api-access-sb6nd\" (UniqueName: \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-kube-api-access-sb6nd\") pod \"da3aab38-b02f-45af-8784-33dd8a2522c2\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") "
Nov 01 04:09:11.500759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.500763    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-certs\") pod \"da3aab38-b02f-45af-8784-33dd8a2522c2\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") "
Nov 01 04:09:11.501042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.500803    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-socket\") pod \"da3aab38-b02f-45af-8784-33dd8a2522c2\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") "
Nov 01 04:09:11.501042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.500848    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-podinfo\") pod \"da3aab38-b02f-45af-8784-33dd8a2522c2\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") "
Nov 01 04:09:11.501042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.500978    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-token\" (UniqueName: \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-token\") pod \"da3aab38-b02f-45af-8784-33dd8a2522c2\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") "
Nov 01 04:09:11.501042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.501031    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-envoy\") pod \"da3aab38-b02f-45af-8784-33dd8a2522c2\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") "
Nov 01 04:09:11.501252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.501070    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-certs" pod "da3aab38-b02f-45af-8784-33dd8a2522c2" (UID: "da3aab38-b02f-45af-8784-33dd8a2522c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:09:11.501252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.501123    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-socket" pod "da3aab38-b02f-45af-8784-33dd8a2522c2" (UID: "da3aab38-b02f-45af-8784-33dd8a2522c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:09:11.501252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.501202    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3aab38-b02f-45af-8784-33dd8a2522c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3aab38-b02f-45af-8784-33dd8a2522c2-istiod-ca-cert\") pod \"da3aab38-b02f-45af-8784-33dd8a2522c2\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") "
Nov 01 04:09:11.501403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.501332    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-data\") pod \"da3aab38-b02f-45af-8784-33dd8a2522c2\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") "
Nov 01 04:09:11.501403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.501361    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-credential-socket\") pod \"da3aab38-b02f-45af-8784-33dd8a2522c2\" (UID: \"da3aab38-b02f-45af-8784-33dd8a2522c2\") "
Nov 01 04:09:11.501580 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.501548    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-data" pod "da3aab38-b02f-45af-8784-33dd8a2522c2" (UID: "da3aab38-b02f-45af-8784-33dd8a2522c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:09:11.501648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.501598    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3aab38-b02f-45af-8784-33dd8a2522c2-istiod-ca-cert" pod "da3aab38-b02f-45af-8784-33dd8a2522c2" (UID: "da3aab38-b02f-45af-8784-33dd8a2522c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:09:11.501854 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.501813    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-credential-socket" pod "da3aab38-b02f-45af-8784-33dd8a2522c2" (UID: "da3aab38-b02f-45af-8784-33dd8a2522c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:09:11.502114 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.502088    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da3aab38-b02f-45af-8784-33dd8a2522c2-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:11.502191 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.502120    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:11.502191 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.502137    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:11.502191 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.502152    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:11.502191 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.502168    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:11.503349 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.503311    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-podinfo" pod "da3aab38-b02f-45af-8784-33dd8a2522c2" (UID: "da3aab38-b02f-45af-8784-33dd8a2522c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:09:11.503497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.503358    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-envoy" pod "da3aab38-b02f-45af-8784-33dd8a2522c2" (UID: "da3aab38-b02f-45af-8784-33dd8a2522c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:09:11.503497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.503387    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-kube-api-access-sb6nd" pod "da3aab38-b02f-45af-8784-33dd8a2522c2" (UID: "da3aab38-b02f-45af-8784-33dd8a2522c2"). InnerVolumeSpecName "kube-api-access-sb6nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:09:11.503497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.503407    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-token" pod "da3aab38-b02f-45af-8784-33dd8a2522c2" (UID: "da3aab38-b02f-45af-8784-33dd8a2522c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:09:11.604000 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.603968    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:11.604239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.604223    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:11.604343 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.604331    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb6nd\" (UniqueName: \"kubernetes.io/projected/da3aab38-b02f-45af-8784-33dd8a2522c2-kube-api-access-sb6nd\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:11.604443 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.604415    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da3aab38-b02f-45af-8784-33dd8a2522c2-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:11.634080 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.634052    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:09:11.635144 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.635119    4882 patch_prober.go:29] interesting pod/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 04:09:11.635194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.635172    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 04:09:11.804040 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.804005    4882 generic.go:356] "Generic (PLEG): container finished" podID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerID="7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a" exitCode=0
Nov 01 04:09:11.804526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.804053    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"
Nov 01 04:09:11.804526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.804085    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" event={"ID":"da3aab38-b02f-45af-8784-33dd8a2522c2","Type":"ContainerDied","Data":"7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a"}
Nov 01 04:09:11.804526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.804149    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc" event={"ID":"da3aab38-b02f-45af-8784-33dd8a2522c2","Type":"ContainerDied","Data":"91e66683f6512db1a60df414ccb4181011a98c3ce7a1afaff2aadd1a2214aea4"}
Nov 01 04:09:11.804526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.804178    4882 scope.go:122] "RemoveContainer" containerID="7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a"
Nov 01 04:09:11.815284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.815263    4882 scope.go:122] "RemoveContainer" containerID="7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a"
Nov 01 04:09:11.815612 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:09:11.815593    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a\": container with ID starting with 7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a not found: ID does not exist" containerID="7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a"
Nov 01 04:09:11.815665 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.815627    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a"} err="failed to get container status \"7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a\": rpc error: code = NotFound desc = could not find container \"7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a\": container with ID starting with 7776ad554a1282ae91e5c7c7838358151f2bf7c69a83257968d98ccb6b12cb9a not found: ID does not exist"
Nov 01 04:09:11.959132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.959102    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"]
Nov 01 04:09:11.984859 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:11.984166    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-w6dgc"]
Nov 01 04:09:12.512648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:12.512606    4882 scope.go:122] "RemoveContainer" containerID="48463e31610292fac624333bfe12f413010faa963957ed80ca4917f4f8fc144a"
Nov 01 04:09:12.524148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:12.524129    4882 scope.go:122] "RemoveContainer" containerID="3ce6856118ea537c1fc3c69c72728e1bed57ceac015e07263379d213a4984396"
Nov 01 04:09:12.534150 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:12.534127    4882 scope.go:122] "RemoveContainer" containerID="3f0cef7e0b76be0c446d68b1dd97f2aa1baa22b26d51dcad877a539ef5262698"
Nov 01 04:09:12.544199 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:12.544177    4882 scope.go:122] "RemoveContainer" containerID="ae2ed16beaf71c016634059288c0ff991e05cf35d0913a62dcb0f439d8fb427f"
Nov 01 04:09:12.553619 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:12.553596    4882 scope.go:122] "RemoveContainer" containerID="ce00eebb4dcab8222d0d360ded8ae1251e5b0ca262a3587a430b6b03e9fccfc7"
Nov 01 04:09:12.594335 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:12.594311    4882 scope.go:122] "RemoveContainer" containerID="9fab6f6f5c394793d705e014ec4721c809716ce6340b202fada28e07b29ac2da"
Nov 01 04:09:12.604857 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:12.604831    4882 scope.go:122] "RemoveContainer" containerID="fd5f8bd4732ccb574b07973a0ad1a966e94dbafb37be9f0955903351b47d2709"
Nov 01 04:09:12.615422 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:12.615401    4882 scope.go:122] "RemoveContainer" containerID="526abc36c3e6ed7538b2dae4a751f575dd8428aad0e63c6d892f05da2ae5aed0"
Nov 01 04:09:12.625388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:12.625358    4882 scope.go:122] "RemoveContainer" containerID="702c735919305e9b956193074ccbd73a7fe091a3aebf5dea346ee1f6d433539a"
Nov 01 04:09:12.634890 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:12.634868    4882 patch_prober.go:29] interesting pod/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 04:09:12.634993 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:12.634942    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 04:09:13.601729 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:13.601696    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" path="/var/lib/kubelet/pods/da3aab38-b02f-45af-8784-33dd8a2522c2/volumes"
Nov 01 04:09:13.634622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:13.634587    4882 patch_prober.go:29] interesting pod/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 04:09:13.634785 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:13.634643    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 04:09:14.634242 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:14.634208    4882 patch_prober.go:29] interesting pod/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 04:09:14.634697 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:14.634265    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 04:09:14.851284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:14.851245    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:09:15.634385 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:15.634349    4882 patch_prober.go:29] interesting pod/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 04:09:15.634824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:15.634403    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 04:09:16.639204 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:16.639167    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:09:16.640325 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:16.640301    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:09:28.724301 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:09:28.724257    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:09:34.164931 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:34.164886    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"]
Nov 01 04:09:34.211350 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:34.211312    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aab51f-p7rgt"]
Nov 01 04:09:35.600694 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:35.600650    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4805e0cb-7874-4741-a266-cc554fcc3391" path="/var/lib/kubelet/pods/4805e0cb-7874-4741-a266-cc554fcc3391/volumes"
Nov 01 04:09:41.815265 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.815221    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"]
Nov 01 04:09:41.816033 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816009    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy"
Nov 01 04:09:41.816143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816033    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy"
Nov 01 04:09:41.816143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816054    4882 cpu_manager.go:414] "containerMap: removing container" podUID="551c4d53-6477-4592-99c4-486fd57ba643" containerName="istio-proxy"
Nov 01 04:09:41.816143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816061    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551c4d53-6477-4592-99c4-486fd57ba643" containerName="istio-proxy"
Nov 01 04:09:41.816143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816080    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerName="istio-proxy"
Nov 01 04:09:41.816143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816088    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerName="istio-proxy"
Nov 01 04:09:41.816143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816100    4882 cpu_manager.go:414] "containerMap: removing container" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy"
Nov 01 04:09:41.816143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816107    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy"
Nov 01 04:09:41.816143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816137    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6c799659-ba33-4ce6-826c-58897edd2134" containerName="istio-proxy"
Nov 01 04:09:41.816143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816144    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c799659-ba33-4ce6-826c-58897edd2134" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816163    4882 cpu_manager.go:414] "containerMap: removing container" podUID="03c4eea3-e198-46d1-adb5-f74a2d8ed561" containerName="revoke-keys"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816170    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c4eea3-e198-46d1-adb5-f74a2d8ed561" containerName="revoke-keys"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816197    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816204    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816224    4882 cpu_manager.go:414] "containerMap: removing container" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816231    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816255    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816262    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816410    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c799659-ba33-4ce6-826c-58897edd2134" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816442    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c4eea3-e198-46d1-adb5-f74a2d8ed561" containerName="revoke-keys"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816457    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816468    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a77365-4bae-4d90-95c9-43f6eb75d25a" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816479    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="948bb297-3490-4419-9cbd-5e11c367c538" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816491    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d36b171-563c-4230-b2fc-aa810dceaa0b" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816500    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816509    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="551c4d53-6477-4592-99c4-486fd57ba643" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816519    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fed9c9-bba5-4025-a338-8d7c7a831941" containerName="istio-proxy"
Nov 01 04:09:41.816703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.816527    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3aab38-b02f-45af-8784-33dd8a2522c2" containerName="istio-proxy"
Nov 01 04:09:41.821627 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.821605    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:41.873778 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.873739    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"]
Nov 01 04:09:41.915696 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.915652    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:09:41.915696 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.915667    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:09:41.959040 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.959001    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6dd36b-8595k\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:41.959226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.959069    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6dd36b-8595k\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:41.959349 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:41.959323    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2sx9\" (UniqueName: \"kubernetes.io/projected/decbaecd-a490-4d3e-bb88-75188792bb92-kube-api-access-r2sx9\") pod \"maas-api-revoke-keys-e2e-api-b-6dd36b-8595k\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:42.060578 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:42.060535    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6dd36b-8595k\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:42.060767 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:42.060725    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2sx9\" (UniqueName: \"kubernetes.io/projected/decbaecd-a490-4d3e-bb88-75188792bb92-kube-api-access-r2sx9\") pod \"maas-api-revoke-keys-e2e-api-b-6dd36b-8595k\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:42.060767 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:42.060758    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6dd36b-8595k\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:42.061219 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:42.061200    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6dd36b-8595k\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:42.063106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:42.063088    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6dd36b-8595k\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:42.117687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:42.117605    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2sx9\" (UniqueName: \"kubernetes.io/projected/decbaecd-a490-4d3e-bb88-75188792bb92-kube-api-access-r2sx9\") pod \"maas-api-revoke-keys-e2e-api-b-6dd36b-8595k\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:42.135333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:42.135299    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:42.511964 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:42.511930    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"]
Nov 01 04:09:42.518651 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:09:42.518611    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecbaecd_a490_4d3e_bb88_75188792bb92.slice/crio-c123a452d5ce0d2ad7f4b50bae35c06df243b0038fe5289d9caa512d8ef03895 WatchSource:0}: Error finding container c123a452d5ce0d2ad7f4b50bae35c06df243b0038fe5289d9caa512d8ef03895: Status 404 returned error can't find the container with id c123a452d5ce0d2ad7f4b50bae35c06df243b0038fe5289d9caa512d8ef03895
Nov 01 04:09:43.152026 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:43.151934    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k" event={"ID":"decbaecd-a490-4d3e-bb88-75188792bb92","Type":"ContainerStarted","Data":"6d87962f98a5fc82576e9bae1384fc291dfb93ebf335f03ed05b9a2c1e249ccb"}
Nov 01 04:09:43.152026 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:43.151994    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k" event={"ID":"decbaecd-a490-4d3e-bb88-75188792bb92","Type":"ContainerStarted","Data":"c123a452d5ce0d2ad7f4b50bae35c06df243b0038fe5289d9caa512d8ef03895"}
Nov 01 04:09:43.262508 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:43.262450    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k" podStartSLOduration=2.262416863 podStartE2EDuration="2.262416863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:09:43.258893316 +0000 UTC m=+1651.820926334" watchObservedRunningTime="2026-11-01 04:09:43.262416863 +0000 UTC m=+1651.824449896"
Nov 01 04:09:44.160040 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:44.160001    4882 generic.go:356] "Generic (PLEG): container finished" podID="decbaecd-a490-4d3e-bb88-75188792bb92" containerID="6d87962f98a5fc82576e9bae1384fc291dfb93ebf335f03ed05b9a2c1e249ccb" exitCode=0
Nov 01 04:09:44.160481 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:44.160128    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k" event={"ID":"decbaecd-a490-4d3e-bb88-75188792bb92","Type":"ContainerDied","Data":"6d87962f98a5fc82576e9bae1384fc291dfb93ebf335f03ed05b9a2c1e249ccb"}
Nov 01 04:09:45.044887 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.044852    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"]
Nov 01 04:09:45.090962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.090917    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8cb6dac-bwrkv"]
Nov 01 04:09:45.326660 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.326632    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:45.505295 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.505258    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decbaecd-a490-4d3e-bb88-75188792bb92-kube-api-access-r2sx9\" (UniqueName: \"kubernetes.io/projected/decbaecd-a490-4d3e-bb88-75188792bb92-kube-api-access-r2sx9\") pod \"decbaecd-a490-4d3e-bb88-75188792bb92\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") "
Nov 01 04:09:45.505538 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.505327    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-openshift-ca\") pod \"decbaecd-a490-4d3e-bb88-75188792bb92\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") "
Nov 01 04:09:45.505538 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.505388    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-xks-ca\") pod \"decbaecd-a490-4d3e-bb88-75188792bb92\" (UID: \"decbaecd-a490-4d3e-bb88-75188792bb92\") "
Nov 01 04:09:45.505762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.505738    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-openshift-ca" pod "decbaecd-a490-4d3e-bb88-75188792bb92" (UID: "decbaecd-a490-4d3e-bb88-75188792bb92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:09:45.505869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.505853    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:45.507506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.507479    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-xks-ca" pod "decbaecd-a490-4d3e-bb88-75188792bb92" (UID: "decbaecd-a490-4d3e-bb88-75188792bb92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:09:45.507620 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.507528    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decbaecd-a490-4d3e-bb88-75188792bb92-kube-api-access-r2sx9" pod "decbaecd-a490-4d3e-bb88-75188792bb92" (UID: "decbaecd-a490-4d3e-bb88-75188792bb92"). InnerVolumeSpecName "kube-api-access-r2sx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:09:45.600483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.600379    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004ac357-ceb3-499a-903c-1204fce4a504" path="/var/lib/kubelet/pods/004ac357-ceb3-499a-903c-1204fce4a504/volumes"
Nov 01 04:09:45.606671 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.606639    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decbaecd-a490-4d3e-bb88-75188792bb92-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:45.606671 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:45.606668    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2sx9\" (UniqueName: \"kubernetes.io/projected/decbaecd-a490-4d3e-bb88-75188792bb92-kube-api-access-r2sx9\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:46.174991 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:46.174953    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"
Nov 01 04:09:46.174991 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:46.174981    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k" event={"ID":"decbaecd-a490-4d3e-bb88-75188792bb92","Type":"ContainerDied","Data":"c123a452d5ce0d2ad7f4b50bae35c06df243b0038fe5289d9caa512d8ef03895"}
Nov 01 04:09:46.175211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:46.175033    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c123a452d5ce0d2ad7f4b50bae35c06df243b0038fe5289d9caa512d8ef03895"
Nov 01 04:09:56.088215 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.088072    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"]
Nov 01 04:09:56.088712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.088697    4882 cpu_manager.go:414] "containerMap: removing container" podUID="decbaecd-a490-4d3e-bb88-75188792bb92" containerName="revoke-keys"
Nov 01 04:09:56.088758 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.088714    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decbaecd-a490-4d3e-bb88-75188792bb92" containerName="revoke-keys"
Nov 01 04:09:56.088758 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.088745    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy"
Nov 01 04:09:56.088758 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.088751    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f68cfc4-6033-4820-bb68-e19695d59202" containerName="istio-proxy"
Nov 01 04:09:56.088870 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.088857    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="decbaecd-a490-4d3e-bb88-75188792bb92" containerName="revoke-keys"
Nov 01 04:09:56.093408 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.093387    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:56.145322 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.145284    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:56.145525 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.145357    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:56.145525 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.145412    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdnh7\" (UniqueName: \"kubernetes.io/projected/83d6a349-b4d5-4f02-a0a4-104ac498ff51-kube-api-access-bdnh7\") pod \"maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:56.188164 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.188131    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:09:56.188328 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.188223    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:09:56.191700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.191670    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"]
Nov 01 04:09:56.246246 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.246199    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:56.246499 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.246267    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:56.246499 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.246322    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdnh7\" (UniqueName: \"kubernetes.io/projected/83d6a349-b4d5-4f02-a0a4-104ac498ff51-kube-api-access-bdnh7\") pod \"maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:56.247047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.247027    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:56.248793 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.248774    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:56.303947 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.303918    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdnh7\" (UniqueName: \"kubernetes.io/projected/83d6a349-b4d5-4f02-a0a4-104ac498ff51-kube-api-access-bdnh7\") pod \"maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:56.407606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.407501    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:56.786819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:56.786781    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"]
Nov 01 04:09:56.793763 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:09:56.793720    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d6a349_b4d5_4f02_a0a4_104ac498ff51.slice/crio-4c7a43d08aecc86ed7e5e655b8f7a795e223d00fe69bc48474d8e4aca1734e21 WatchSource:0}: Error finding container 4c7a43d08aecc86ed7e5e655b8f7a795e223d00fe69bc48474d8e4aca1734e21: Status 404 returned error can't find the container with id 4c7a43d08aecc86ed7e5e655b8f7a795e223d00fe69bc48474d8e4aca1734e21
Nov 01 04:09:57.309229 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:57.309177    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz" event={"ID":"83d6a349-b4d5-4f02-a0a4-104ac498ff51","Type":"ContainerStarted","Data":"2b7bd92051b910c5fcc7260e2e3c262565662ada44b21d951766956b71ad7541"}
Nov 01 04:09:57.309757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:57.309295    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz" event={"ID":"83d6a349-b4d5-4f02-a0a4-104ac498ff51","Type":"ContainerStarted","Data":"4c7a43d08aecc86ed7e5e655b8f7a795e223d00fe69bc48474d8e4aca1734e21"}
Nov 01 04:09:57.421048 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:57.420938    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz" podStartSLOduration=1.4209224200000001 podStartE2EDuration="1.42092242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:09:57.418191786 +0000 UTC m=+1665.980224804" watchObservedRunningTime="2026-11-01 04:09:57.42092242 +0000 UTC m=+1665.982955463"
Nov 01 04:09:58.317250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:58.317212    4882 generic.go:356] "Generic (PLEG): container finished" podID="83d6a349-b4d5-4f02-a0a4-104ac498ff51" containerID="2b7bd92051b910c5fcc7260e2e3c262565662ada44b21d951766956b71ad7541" exitCode=0
Nov 01 04:09:58.317704 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:58.317300    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz" event={"ID":"83d6a349-b4d5-4f02-a0a4-104ac498ff51","Type":"ContainerDied","Data":"2b7bd92051b910c5fcc7260e2e3c262565662ada44b21d951766956b71ad7541"}
Nov 01 04:09:58.724645 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:09:58.724607    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:09:59.497069 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:59.497030    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:09:59.590183 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:59.590105    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d6a349-b4d5-4f02-a0a4-104ac498ff51-kube-api-access-bdnh7\" (UniqueName: \"kubernetes.io/projected/83d6a349-b4d5-4f02-a0a4-104ac498ff51-kube-api-access-bdnh7\") pod \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") "
Nov 01 04:09:59.590490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:59.590311    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-openshift-ca\") pod \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") "
Nov 01 04:09:59.590490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:59.590363    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-xks-ca\") pod \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\" (UID: \"83d6a349-b4d5-4f02-a0a4-104ac498ff51\") "
Nov 01 04:09:59.590694 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:59.590669    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-openshift-ca" pod "83d6a349-b4d5-4f02-a0a4-104ac498ff51" (UID: "83d6a349-b4d5-4f02-a0a4-104ac498ff51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:09:59.590969 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:59.590950    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:59.593045 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:59.593015    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-xks-ca" pod "83d6a349-b4d5-4f02-a0a4-104ac498ff51" (UID: "83d6a349-b4d5-4f02-a0a4-104ac498ff51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:09:59.593183 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:59.593152    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d6a349-b4d5-4f02-a0a4-104ac498ff51-kube-api-access-bdnh7" pod "83d6a349-b4d5-4f02-a0a4-104ac498ff51" (UID: "83d6a349-b4d5-4f02-a0a4-104ac498ff51"). InnerVolumeSpecName "kube-api-access-bdnh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:09:59.692812 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:59.692690    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdnh7\" (UniqueName: \"kubernetes.io/projected/83d6a349-b4d5-4f02-a0a4-104ac498ff51-kube-api-access-bdnh7\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:09:59.692812 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:09:59.692730    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d6a349-b4d5-4f02-a0a4-104ac498ff51-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:00.334732 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:00.334702    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"
Nov 01 04:10:00.334912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:00.334740    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz" event={"ID":"83d6a349-b4d5-4f02-a0a4-104ac498ff51","Type":"ContainerDied","Data":"4c7a43d08aecc86ed7e5e655b8f7a795e223d00fe69bc48474d8e4aca1734e21"}
Nov 01 04:10:00.334912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:00.334797    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c7a43d08aecc86ed7e5e655b8f7a795e223d00fe69bc48474d8e4aca1734e21"
Nov 01 04:10:01.044237 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:01.044202    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"]
Nov 01 04:10:01.090641 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:01.090595    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dc88a5-t42t5"]
Nov 01 04:10:01.600714 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:01.600680    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348" path="/var/lib/kubelet/pods/7a5f4e3a-8889-4e9d-88b9-01e1c7ffb348/volumes"
Nov 01 04:10:12.827576 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:12.827537    4882 scope.go:122] "RemoveContainer" containerID="610d888314db586389efa3c7cae57496d793754e141031fbebc4f26f54e9e045"
Nov 01 04:10:12.837968 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:12.837935    4882 scope.go:122] "RemoveContainer" containerID="0d1cef3c63cc8cff4d7c23bdd1d9fc2694e4e187be4b5a9016ddd636c0dbba23"
Nov 01 04:10:26.522363 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.522326    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"]
Nov 01 04:10:26.522933 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.522917    4882 cpu_manager.go:414] "containerMap: removing container" podUID="83d6a349-b4d5-4f02-a0a4-104ac498ff51" containerName="revoke-keys"
Nov 01 04:10:26.522981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.522934    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d6a349-b4d5-4f02-a0a4-104ac498ff51" containerName="revoke-keys"
Nov 01 04:10:26.523030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.523021    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d6a349-b4d5-4f02-a0a4-104ac498ff51" containerName="revoke-keys"
Nov 01 04:10:26.528030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.528010    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.708899 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.708859    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5xvf\" (UniqueName: \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-kube-api-access-g5xvf\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.709100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.709021    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.709100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.709088    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.709204 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.709156    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.709304 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.709280    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.709365 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.709314    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.709477 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.709414    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.709536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.709494    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.709536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.709525    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.726653 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.726617    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"]
Nov 01 04:10:26.811240 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811126    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.811240 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811182    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.811533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811283    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.811533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811309    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.811533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811371    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.811533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811401    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.811533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811479    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.811788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811609    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5xvf\" (UniqueName: \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-kube-api-access-g5xvf\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.811788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811664    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.811788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811679    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.811788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811701    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.811788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811769    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.812051 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.811913    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.812388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.812364    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.813905 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.813875    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.814155 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.814134    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.868697 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.868660    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:26.868853 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:26.868800    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5xvf\" (UniqueName: \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-kube-api-access-g5xvf\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:27.144345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:27.144256    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:27.581749 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:27.581712    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"]
Nov 01 04:10:27.586760 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:10:27.586732    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode426a233_79fe_4e5e_b282_7e1c4cfe7058.slice/crio-fd1c9b4ba914eb186e58d455633bf830e4f74ec24bf210fede001eb5fec53d33 WatchSource:0}: Error finding container fd1c9b4ba914eb186e58d455633bf830e4f74ec24bf210fede001eb5fec53d33: Status 404 returned error can't find the container with id fd1c9b4ba914eb186e58d455633bf830e4f74ec24bf210fede001eb5fec53d33
Nov 01 04:10:27.589354 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:27.589318    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:10:27.589483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:27.589385    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:10:27.589483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:27.589412    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:10:28.554598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:28.554562    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" event={"ID":"e426a233-79fe-4e5e-b282-7e1c4cfe7058","Type":"ContainerStarted","Data":"5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29"}
Nov 01 04:10:28.554792 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:28.554619    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" event={"ID":"e426a233-79fe-4e5e-b282-7e1c4cfe7058","Type":"ContainerStarted","Data":"fd1c9b4ba914eb186e58d455633bf830e4f74ec24bf210fede001eb5fec53d33"}
Nov 01 04:10:28.714716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:28.714664    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" podStartSLOduration=2.71464397 podStartE2EDuration="2.71464397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:10:28.713649647 +0000 UTC m=+1697.275682660" watchObservedRunningTime="2026-11-01 04:10:28.71464397 +0000 UTC m=+1697.276676988"
Nov 01 04:10:28.735358 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:10:28.735312    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:10:29.145438 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:29.145394    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:29.147171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:29.147141    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 04:10:29.147320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:29.147199    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" podUID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 04:10:30.145504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:30.145466    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 04:10:30.145921 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:30.145553    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" podUID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 04:10:31.144905 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:31.144869    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 04:10:31.145236 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:31.144925    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" podUID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 04:10:32.144893 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.144854    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 04:10:32.145399 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.144924    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" podUID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 04:10:32.456692 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.456655    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"]
Nov 01 04:10:32.460626 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.460603    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.505265 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.505148    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea856e65-e162-4031-ac33-acf56ce4b296-tls-certs\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.505478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.505284    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-tmp-dir\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.505478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.505403    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-model-cache\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.505633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.505503    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg626\" (UniqueName: \"kubernetes.io/projected/ea856e65-e162-4031-ac33-acf56ce4b296-kube-api-access-wg626\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.505633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.505583    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-dshm\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.505756 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.505654    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-home\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.558243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.557701    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4adf4c\"/\"openshift-service-ca.crt\""
Nov 01 04:10:32.558243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.557738    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4adf4c\"/\"default-dockercfg-98mlh\""
Nov 01 04:10:32.558243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.557738    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4adf4c\"/\"tenantcfg-model-4adf4c-kserve-self-signed-certs\""
Nov 01 04:10:32.558243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.557847    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4adf4c\"/\"kube-root-ca.crt\""
Nov 01 04:10:32.587577 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.586493    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"]
Nov 01 04:10:32.616518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.615423    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-home\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.616518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.615562    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea856e65-e162-4031-ac33-acf56ce4b296-tls-certs\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.616518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.615607    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-tmp-dir\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.616518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.615721    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-model-cache\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.616518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.615854    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg626\" (UniqueName: \"kubernetes.io/projected/ea856e65-e162-4031-ac33-acf56ce4b296-kube-api-access-wg626\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.616518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.615891    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-dshm\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.617066 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.617032    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-tmp-dir\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.618228 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.617339    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-home\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.618588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.618564    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-model-cache\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.621081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.621056    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea856e65-e162-4031-ac33-acf56ce4b296-tls-certs\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.624031 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.623942    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-dshm\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.697360 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.697326    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg626\" (UniqueName: \"kubernetes.io/projected/ea856e65-e162-4031-ac33-acf56ce4b296-kube-api-access-wg626\") pod \"tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") " pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:32.799345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:32.799257    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:33.145019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:33.144926    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 04:10:33.145019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:33.144989    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" podUID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 04:10:33.189454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:33.183146    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"]
Nov 01 04:10:33.625312 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:33.625266    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj" event={"ID":"ea856e65-e162-4031-ac33-acf56ce4b296","Type":"ContainerStarted","Data":"90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e"}
Nov 01 04:10:33.625543 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:33.625320    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj" event={"ID":"ea856e65-e162-4031-ac33-acf56ce4b296","Type":"ContainerStarted","Data":"8f1eb554a9f40a7174dbbf09abeb8a034fb9cfc4ed3959b3d677bade539ada72"}
Nov 01 04:10:33.746772 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:33.746479    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj" podStartSLOduration=1.746458301 podStartE2EDuration="1.746458301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:10:33.741117035 +0000 UTC m=+1702.303150057" watchObservedRunningTime="2026-11-01 04:10:33.746458301 +0000 UTC m=+1702.308491315"
Nov 01 04:10:33.800851 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:33.799499    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:34.145026 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:34.144989    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 04:10:34.145472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:34.145059    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" podUID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 04:10:35.150035 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:35.150004    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:35.150459 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:35.150438    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:35.151092 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:35.151071    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:10:41.538147 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.538104    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"]
Nov 01 04:10:41.550784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.550750    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.648593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.648541    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"]
Nov 01 04:10:41.665394 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.665335    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.665653 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.665530    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.665653 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.665567    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.665789 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.665729    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.665872 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.665851    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8ps\" (UniqueName: \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-kube-api-access-jt8ps\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.665944 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.665903    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.666002 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.665969    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.666058 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.666009    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.666058 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.666036    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767039    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767270 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767120    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767270 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767212    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767270 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767243    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767484 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767320    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767484 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767417    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8ps\" (UniqueName: \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-kube-api-access-jt8ps\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767484 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767477    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767650 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767543    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767650 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767584    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767650 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767634    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767810 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767680    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767810 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767721    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.767979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.767959    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.768237 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.768219    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.769632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.769607    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.769737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.769721    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.824898 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.824817    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.824898 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.824886    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8ps\" (UniqueName: \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-kube-api-access-jt8ps\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:41.883885 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:41.883804    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:42.052912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.052163    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"]
Nov 01 04:10:42.058732 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.058661    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-907dc852-mx9r4"]
Nov 01 04:10:42.342636 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.342600    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"]
Nov 01 04:10:42.347490 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:10:42.347421    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a0c5ff_b00b_4ca8_9d58_50c4610e23d5.slice/crio-d4242ebd9fb15294ab9c646eb6dd0c0677bba4d7a2716375b1d73445f6eb1a31 WatchSource:0}: Error finding container d4242ebd9fb15294ab9c646eb6dd0c0677bba4d7a2716375b1d73445f6eb1a31: Status 404 returned error can't find the container with id d4242ebd9fb15294ab9c646eb6dd0c0677bba4d7a2716375b1d73445f6eb1a31
Nov 01 04:10:42.349861 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.349829    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:10:42.349934 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.349901    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:10:42.349934 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.349929    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:10:42.692334 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.692299    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk" event={"ID":"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5","Type":"ContainerStarted","Data":"4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2"}
Nov 01 04:10:42.692869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.692355    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk" event={"ID":"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5","Type":"ContainerStarted","Data":"d4242ebd9fb15294ab9c646eb6dd0c0677bba4d7a2716375b1d73445f6eb1a31"}
Nov 01 04:10:42.800037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.799999    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:42.813592 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.813562    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:42.825657 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.825582    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk" podStartSLOduration=1.825563191 podStartE2EDuration="1.825563191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:10:42.823832898 +0000 UTC m=+1711.385865915" watchObservedRunningTime="2026-11-01 04:10:42.825563191 +0000 UTC m=+1711.387596209"
Nov 01 04:10:42.885451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:42.884519    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:43.390148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:43.390109    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"]
Nov 01 04:10:43.395423 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:43.395390    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3344e8-jjz6q"]
Nov 01 04:10:43.607493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:43.607457    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c4eea3-e198-46d1-adb5-f74a2d8ed561" path="/var/lib/kubelet/pods/03c4eea3-e198-46d1-adb5-f74a2d8ed561/volumes"
Nov 01 04:10:43.607969 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:43.607955    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3eaac9-81e1-433a-ac79-ef1bbc8c773d" path="/var/lib/kubelet/pods/3e3eaac9-81e1-433a-ac79-ef1bbc8c773d/volumes"
Nov 01 04:10:43.710157 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:43.710128    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:10:43.888904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:43.888859    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 01 04:10:43.889112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:43.888926    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk" podUID="03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 01 04:10:43.891232 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:43.891200    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"]
Nov 01 04:10:43.901301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:43.901261    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:43.917699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:43.917608    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"]
Nov 01 04:10:44.018916 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.018828    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-fkplf\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:44.019075 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.018949    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jt2s\" (UniqueName: \"kubernetes.io/projected/aa974e26-5c74-422c-a159-a1659b89f3c0-kube-api-access-9jt2s\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-fkplf\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:44.019075 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.019026    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-fkplf\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:44.022111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.022084    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:10:44.022231 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.022159    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:10:44.120647 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.120614    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-fkplf\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:44.120845 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.120668    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jt2s\" (UniqueName: \"kubernetes.io/projected/aa974e26-5c74-422c-a159-a1659b89f3c0-kube-api-access-9jt2s\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-fkplf\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:44.120845 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.120704    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-fkplf\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:44.121497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.121467    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-fkplf\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:44.123096 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.123073    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-fkplf\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:44.187475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.187386    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jt2s\" (UniqueName: \"kubernetes.io/projected/aa974e26-5c74-422c-a159-a1659b89f3c0-kube-api-access-9jt2s\") pod \"maas-api-revoke-keys-e2e-mt-078e0d65-fkplf\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:44.222283 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.222244    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:44.621113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.621021    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"]
Nov 01 04:10:44.627067 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:10:44.627035    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa974e26_5c74_422c_a159_a1659b89f3c0.slice/crio-7e99301afcd870b2a831689ce1a0b18aaef4e2a56fe641a875f02a905617cbc7 WatchSource:0}: Error finding container 7e99301afcd870b2a831689ce1a0b18aaef4e2a56fe641a875f02a905617cbc7: Status 404 returned error can't find the container with id 7e99301afcd870b2a831689ce1a0b18aaef4e2a56fe641a875f02a905617cbc7
Nov 01 04:10:44.706926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.706885    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf" event={"ID":"aa974e26-5c74-422c-a159-a1659b89f3c0","Type":"ContainerStarted","Data":"7e99301afcd870b2a831689ce1a0b18aaef4e2a56fe641a875f02a905617cbc7"}
Nov 01 04:10:44.888307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.888209    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 01 04:10:44.888307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:44.888267    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk" podUID="03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 01 04:10:45.716028 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:45.715993    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf" event={"ID":"aa974e26-5c74-422c-a159-a1659b89f3c0","Type":"ContainerStarted","Data":"a39bfdcf71bfd7ffe1d3bf9f18fdf1a8c228a8f0353d7f2ca70ac9a5319be084"}
Nov 01 04:10:45.840047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:45.839987    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf" podStartSLOduration=2.839969136 podStartE2EDuration="2.839969136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:10:45.837037286 +0000 UTC m=+1714.399070302" watchObservedRunningTime="2026-11-01 04:10:45.839969136 +0000 UTC m=+1714.402002152"
Nov 01 04:10:45.886658 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:45.886608    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 01 04:10:45.886838 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:45.886708    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk" podUID="03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 01 04:10:46.725236 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:46.725198    4882 generic.go:356] "Generic (PLEG): container finished" podID="aa974e26-5c74-422c-a159-a1659b89f3c0" containerID="a39bfdcf71bfd7ffe1d3bf9f18fdf1a8c228a8f0353d7f2ca70ac9a5319be084" exitCode=0
Nov 01 04:10:46.725656 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:46.725277    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf" event={"ID":"aa974e26-5c74-422c-a159-a1659b89f3c0","Type":"ContainerDied","Data":"a39bfdcf71bfd7ffe1d3bf9f18fdf1a8c228a8f0353d7f2ca70ac9a5319be084"}
Nov 01 04:10:46.885901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:46.885861    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 01 04:10:46.886192 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:46.886162    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk" podUID="03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 01 04:10:47.893261 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:47.893180    4882 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 01 04:10:47.894159 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:47.894039    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk" podUID="03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 01 04:10:47.932926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:47.932898    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:48.098116 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.098016    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa974e26-5c74-422c-a159-a1659b89f3c0-kube-api-access-9jt2s\" (UniqueName: \"kubernetes.io/projected/aa974e26-5c74-422c-a159-a1659b89f3c0-kube-api-access-9jt2s\") pod \"aa974e26-5c74-422c-a159-a1659b89f3c0\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") "
Nov 01 04:10:48.098305 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.098140    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-openshift-ca\") pod \"aa974e26-5c74-422c-a159-a1659b89f3c0\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") "
Nov 01 04:10:48.098305 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.098261    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-xks-ca\") pod \"aa974e26-5c74-422c-a159-a1659b89f3c0\" (UID: \"aa974e26-5c74-422c-a159-a1659b89f3c0\") "
Nov 01 04:10:48.098719 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.098690    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-openshift-ca" pod "aa974e26-5c74-422c-a159-a1659b89f3c0" (UID: "aa974e26-5c74-422c-a159-a1659b89f3c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:10:48.098939 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.098917    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:48.100797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.100771    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-xks-ca" pod "aa974e26-5c74-422c-a159-a1659b89f3c0" (UID: "aa974e26-5c74-422c-a159-a1659b89f3c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:10:48.102184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.102158    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa974e26-5c74-422c-a159-a1659b89f3c0-kube-api-access-9jt2s" pod "aa974e26-5c74-422c-a159-a1659b89f3c0" (UID: "aa974e26-5c74-422c-a159-a1659b89f3c0"). InnerVolumeSpecName "kube-api-access-9jt2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:10:48.200523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.200482    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa974e26-5c74-422c-a159-a1659b89f3c0-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:48.200523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.200519    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jt2s\" (UniqueName: \"kubernetes.io/projected/aa974e26-5c74-422c-a159-a1659b89f3c0-kube-api-access-9jt2s\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:48.741644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.741607    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf" event={"ID":"aa974e26-5c74-422c-a159-a1659b89f3c0","Type":"ContainerDied","Data":"7e99301afcd870b2a831689ce1a0b18aaef4e2a56fe641a875f02a905617cbc7"}
Nov 01 04:10:48.741644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.741637    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:10:48.741910 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.741661    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e99301afcd870b2a831689ce1a0b18aaef4e2a56fe641a875f02a905617cbc7"
Nov 01 04:10:48.889670 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.889633    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:48.890198 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.890176    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:48.891044 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:48.891015    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:10:50.274970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:50.274932    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"]
Nov 01 04:10:50.275356 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:50.275180    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx" podUID="62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" containerName="istio-proxy" containerID="cri-o://93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff" gracePeriod=30
Nov 01 04:10:51.719700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.719661    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"]
Nov 01 04:10:51.720600 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.720484    4882 cpu_manager.go:414] "containerMap: removing container" podUID="aa974e26-5c74-422c-a159-a1659b89f3c0" containerName="revoke-keys"
Nov 01 04:10:51.720600 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.720509    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa974e26-5c74-422c-a159-a1659b89f3c0" containerName="revoke-keys"
Nov 01 04:10:51.721245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.721224    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa974e26-5c74-422c-a159-a1659b89f3c0" containerName="revoke-keys"
Nov 01 04:10:51.730764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.730550    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.770940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.764324    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-envoy\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.770940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.764547    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-podinfo\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.770940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.764618    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-socket\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.770940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.764752    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2pwc\" (UniqueName: \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-kube-api-access-t2pwc\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.770940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.764808    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-token\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.770940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.764833    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-credential-socket\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.770940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.764917    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istiod-ca-cert\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.770940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.764958    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-data\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.770940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.764990    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-certs\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.867624 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.867143    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-157fed-openshift-default-dockercfg-h8cn5\""
Nov 01 04:10:51.868393 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.868358    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-data\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.868591 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.868570    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-certs\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.870984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.868757    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-envoy\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.870984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.868883    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-podinfo\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.870984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.868938    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-socket\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.870984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.869088    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2pwc\" (UniqueName: \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-kube-api-access-t2pwc\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.870984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.869147    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-token\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.870984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.869172    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-credential-socket\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.870984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.869223    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istiod-ca-cert\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.870984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.870103    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istiod-ca-cert\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.870984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.870386    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-data\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.870984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.870661    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-certs\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.877139 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.876374    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-socket\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.877769 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.877742    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-credential-socket\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.883304 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.883240    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-podinfo\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.922928 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.903152    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-envoy\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.922928 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.903165    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"]
Nov 01 04:10:51.941586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.938413    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2pwc\" (UniqueName: \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-kube-api-access-t2pwc\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:51.941586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:51.938918    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-token\") pod \"e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") " pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:52.063507 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:52.063392    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:52.525208 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:52.525172    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"]
Nov 01 04:10:52.529457 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:10:52.529409    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865370fa_7d36_46e3_98e2_88ca8d35c6b5.slice/crio-137a71980e4a93b0bd8a8a60d6ee1c3d683dbbdd6c17fa116dbb3a2c30514fb2 WatchSource:0}: Error finding container 137a71980e4a93b0bd8a8a60d6ee1c3d683dbbdd6c17fa116dbb3a2c30514fb2: Status 404 returned error can't find the container with id 137a71980e4a93b0bd8a8a60d6ee1c3d683dbbdd6c17fa116dbb3a2c30514fb2
Nov 01 04:10:52.532236 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:52.532198    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:10:52.532330 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:52.532287    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:10:52.532367 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:52.532327    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:10:52.909968 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:52.909880    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" event={"ID":"865370fa-7d36-46e3-98e2-88ca8d35c6b5","Type":"ContainerStarted","Data":"cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01"}
Nov 01 04:10:52.909968 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:52.909943    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" event={"ID":"865370fa-7d36-46e3-98e2-88ca8d35c6b5","Type":"ContainerStarted","Data":"137a71980e4a93b0bd8a8a60d6ee1c3d683dbbdd6c17fa116dbb3a2c30514fb2"}
Nov 01 04:10:53.024395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:53.024330    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" podStartSLOduration=2.024312054 podStartE2EDuration="2.024312054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:10:53.02256345 +0000 UTC m=+1721.584596466" watchObservedRunningTime="2026-11-01 04:10:53.024312054 +0000 UTC m=+1721.586345072"
Nov 01 04:10:53.064125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:53.064086    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:53.070032 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:53.070001    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:53.929448 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:53.928946    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:53.929979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:53.929918    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:10:55.411780 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.411743    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"]
Nov 01 04:10:55.412292 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.412011    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" podUID="f1421745-fcd8-41fc-ba64-ff1336e43684" containerName="istio-proxy" containerID="cri-o://5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb" gracePeriod=30
Nov 01 04:10:55.574091 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.574063    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:10:55.633291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.633249    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-credential-socket\") pod \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") "
Nov 01 04:10:55.633536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.633330    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-kube-api-access-bg4wg\" (UniqueName: \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-kube-api-access-bg4wg\") pod \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") "
Nov 01 04:10:55.633536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.633382    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istiod-ca-cert\") pod \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") "
Nov 01 04:10:55.633536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.633530    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-token\" (UniqueName: \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-token\") pod \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") "
Nov 01 04:10:55.633726 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.633574    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-credential-socket" pod "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" (UID: "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:55.633726 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.633663    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-envoy\") pod \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") "
Nov 01 04:10:55.633726 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.633701    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-socket\") pod \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") "
Nov 01 04:10:55.633880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.633772    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istiod-ca-cert" pod "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" (UID: "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:10:55.633880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.633785    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-data\") pod \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") "
Nov 01 04:10:55.633984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.633888    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-certs\") pod \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") "
Nov 01 04:10:55.633984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.633939    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-podinfo\") pod \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\" (UID: \"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35\") "
Nov 01 04:10:55.634083 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.634000    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-socket" pod "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" (UID: "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:55.634134 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.634104    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-data" pod "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" (UID: "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:55.634642 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.634612    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-certs" pod "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" (UID: "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:55.634771 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.634718    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:55.634771 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.634739    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:55.634771 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.634753    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:55.634771 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.634763    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:55.635897 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.635874    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-kube-api-access-bg4wg" pod "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" (UID: "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35"). InnerVolumeSpecName "kube-api-access-bg4wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:10:55.636129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.636109    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-token" pod "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" (UID: "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:10:55.636310 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.636285    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-envoy" pod "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" (UID: "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:55.636402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.636334    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-podinfo" pod "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" (UID: "62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:10:55.736629 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.736591    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:55.736629 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.736632    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:55.736877 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.736650    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:55.736877 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.736665    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg4wg\" (UniqueName: \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-kube-api-access-bg4wg\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:55.736877 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.736681    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:10:55.972013 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.971972    4882 generic.go:356] "Generic (PLEG): container finished" podID="62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" containerID="93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff" exitCode=0
Nov 01 04:10:55.972198 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.972020    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"
Nov 01 04:10:55.972198 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.972056    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx" event={"ID":"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35","Type":"ContainerDied","Data":"93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff"}
Nov 01 04:10:55.972198 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.972112    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx" event={"ID":"62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35","Type":"ContainerDied","Data":"5745f081b0e9ea627b882e7fc660082b8f741ff93db815086d32eacc58b669ac"}
Nov 01 04:10:55.972198 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.972135    4882 scope.go:122] "RemoveContainer" containerID="93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff"
Nov 01 04:10:55.992059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.992035    4882 scope.go:122] "RemoveContainer" containerID="93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff"
Nov 01 04:10:55.992392 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:10:55.992370    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff\": container with ID starting with 93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff not found: ID does not exist" containerID="93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff"
Nov 01 04:10:55.992479 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:55.992402    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff"} err="failed to get container status \"93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff\": rpc error: code = NotFound desc = could not find container \"93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff\": container with ID starting with 93f060ddf3502f96dcea75be7215f5320e76cc7a7b0649496c727ce40a6893ff not found: ID does not exist"
Nov 01 04:10:56.126902 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:56.126859    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"]
Nov 01 04:10:56.172327 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:56.172277    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-n5xkx"]
Nov 01 04:10:57.600397 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:10:57.600358    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" path="/var/lib/kubelet/pods/62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35/volumes"
Nov 01 04:10:58.725393 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:10:58.725353    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:11:00.773256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.772862    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:11:00.836492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.836410    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-certs\") pod \"f1421745-fcd8-41fc-ba64-ff1336e43684\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") "
Nov 01 04:11:00.836687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.836497    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1421745-fcd8-41fc-ba64-ff1336e43684-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1421745-fcd8-41fc-ba64-ff1336e43684-istiod-ca-cert\") pod \"f1421745-fcd8-41fc-ba64-ff1336e43684\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") "
Nov 01 04:11:00.836687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.836633    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-data\") pod \"f1421745-fcd8-41fc-ba64-ff1336e43684\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") "
Nov 01 04:11:00.836687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.836669    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-kube-api-access-lj2z5\" (UniqueName: \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-kube-api-access-lj2z5\") pod \"f1421745-fcd8-41fc-ba64-ff1336e43684\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") "
Nov 01 04:11:00.836860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.836700    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-podinfo\") pod \"f1421745-fcd8-41fc-ba64-ff1336e43684\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") "
Nov 01 04:11:00.836860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.836734    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-envoy\") pod \"f1421745-fcd8-41fc-ba64-ff1336e43684\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") "
Nov 01 04:11:00.836860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.836772    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-token\" (UniqueName: \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-token\") pod \"f1421745-fcd8-41fc-ba64-ff1336e43684\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") "
Nov 01 04:11:00.836860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.836803    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-credential-socket\") pod \"f1421745-fcd8-41fc-ba64-ff1336e43684\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") "
Nov 01 04:11:00.836860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.836835    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-socket\") pod \"f1421745-fcd8-41fc-ba64-ff1336e43684\" (UID: \"f1421745-fcd8-41fc-ba64-ff1336e43684\") "
Nov 01 04:11:00.838262 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.838230    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-socket" pod "f1421745-fcd8-41fc-ba64-ff1336e43684" (UID: "f1421745-fcd8-41fc-ba64-ff1336e43684"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:11:00.838731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.838702    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-credential-socket" pod "f1421745-fcd8-41fc-ba64-ff1336e43684" (UID: "f1421745-fcd8-41fc-ba64-ff1336e43684"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:11:00.838967 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.838941    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-certs" pod "f1421745-fcd8-41fc-ba64-ff1336e43684" (UID: "f1421745-fcd8-41fc-ba64-ff1336e43684"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:11:00.839284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.839159    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1421745-fcd8-41fc-ba64-ff1336e43684-istiod-ca-cert" pod "f1421745-fcd8-41fc-ba64-ff1336e43684" (UID: "f1421745-fcd8-41fc-ba64-ff1336e43684"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:11:00.839284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.839248    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-data" pod "f1421745-fcd8-41fc-ba64-ff1336e43684" (UID: "f1421745-fcd8-41fc-ba64-ff1336e43684"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:11:00.840498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.840473    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-kube-api-access-lj2z5" pod "f1421745-fcd8-41fc-ba64-ff1336e43684" (UID: "f1421745-fcd8-41fc-ba64-ff1336e43684"). InnerVolumeSpecName "kube-api-access-lj2z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:11:00.840635 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.840536    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-envoy" pod "f1421745-fcd8-41fc-ba64-ff1336e43684" (UID: "f1421745-fcd8-41fc-ba64-ff1336e43684"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:11:00.841995 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.841960    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-podinfo" pod "f1421745-fcd8-41fc-ba64-ff1336e43684" (UID: "f1421745-fcd8-41fc-ba64-ff1336e43684"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:11:00.843382 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.843342    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-token" pod "f1421745-fcd8-41fc-ba64-ff1336e43684" (UID: "f1421745-fcd8-41fc-ba64-ff1336e43684"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:11:00.939057 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.939023    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:00.939057 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.939054    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1421745-fcd8-41fc-ba64-ff1336e43684-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:00.939057 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.939068    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:00.939307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.939083    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj2z5\" (UniqueName: \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-kube-api-access-lj2z5\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:00.939307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.939096    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:00.939307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.939107    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:00.939307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.939118    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1421745-fcd8-41fc-ba64-ff1336e43684-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:00.939307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.939129    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:00.939307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:00.939142    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1421745-fcd8-41fc-ba64-ff1336e43684-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:01.015792 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:01.015702    4882 generic.go:356] "Generic (PLEG): container finished" podID="f1421745-fcd8-41fc-ba64-ff1336e43684" containerID="5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb" exitCode=0
Nov 01 04:11:01.015792 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:01.015752    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"
Nov 01 04:11:01.016050 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:01.015786    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" event={"ID":"f1421745-fcd8-41fc-ba64-ff1336e43684","Type":"ContainerDied","Data":"5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb"}
Nov 01 04:11:01.016050 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:01.015852    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96" event={"ID":"f1421745-fcd8-41fc-ba64-ff1336e43684","Type":"ContainerDied","Data":"e00fd6754e4c72114f3885002cf4178a120b5b11fe3706a84413db316fe8f382"}
Nov 01 04:11:01.016050 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:01.015882    4882 scope.go:122] "RemoveContainer" containerID="5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb"
Nov 01 04:11:01.027319 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:01.027300    4882 scope.go:122] "RemoveContainer" containerID="5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb"
Nov 01 04:11:01.027631 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:11:01.027603    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb\": container with ID starting with 5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb not found: ID does not exist" containerID="5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb"
Nov 01 04:11:01.027685 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:01.027642    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb"} err="failed to get container status \"5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb\": rpc error: code = NotFound desc = could not find container \"5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb\": container with ID starting with 5f7ec1b4b482c99f1adaeb50919d29335d5f15fecf60275461a611f8391b34fb not found: ID does not exist"
Nov 01 04:11:01.171885 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:01.171651    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"]
Nov 01 04:11:01.175500 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:01.175471    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-2lt96"]
Nov 01 04:11:01.602554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:01.602522    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1421745-fcd8-41fc-ba64-ff1336e43684" path="/var/lib/kubelet/pods/f1421745-fcd8-41fc-ba64-ff1336e43684/volumes"
Nov 01 04:11:12.871353 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:12.871317    4882 scope.go:122] "RemoveContainer" containerID="eae55469bfdfa81142e88f5ef0263bf5398d8e01c0065979b965ae145521a65d"
Nov 01 04:11:12.881250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:12.881227    4882 scope.go:122] "RemoveContainer" containerID="21105a863ed20a73d6689c8fbd678a144521d1e8870f3b534ff063b110ee524f"
Nov 01 04:11:21.578701 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:21.578605    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"]
Nov 01 04:11:21.624565 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:21.624530    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"]
Nov 01 04:11:21.643348 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:11:21.643306    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-mt-078e0d65-fkplf\" is forbidden: User \"system:node:ip-10-0-129-252.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-252.us-west-2.compute.internal' and this object" podUID="aa974e26-5c74-422c-a159-a1659b89f3c0" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-078e0d65-fkplf"
Nov 01 04:11:23.601555 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:23.601523    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa974e26-5c74-422c-a159-a1659b89f3c0" path="/var/lib/kubelet/pods/aa974e26-5c74-422c-a159-a1659b89f3c0/volumes"
Nov 01 04:11:28.725422 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:11:28.725378    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:11:29.656317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:29.656279    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"]
Nov 01 04:11:29.656542 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:29.656499    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy" containerID="cri-o://8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b" gracePeriod=30
Nov 01 04:11:33.127452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:33.127388    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"]
Nov 01 04:11:33.173211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:33.173172    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78dad9-wdcgz"]
Nov 01 04:11:33.601885 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:33.601848    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d6a349-b4d5-4f02-a0a4-104ac498ff51" path="/var/lib/kubelet/pods/83d6a349-b4d5-4f02-a0a4-104ac498ff51/volumes"
Nov 01 04:11:34.458058 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.458021    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-jxfk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:11:34.458581 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.458086    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:11:34.933224 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.933194    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 04:11:34.996224 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.996189    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-socket\") pod \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") "
Nov 01 04:11:34.996416 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.996285    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-token\" (UniqueName: \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-token\") pod \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") "
Nov 01 04:11:34.996416 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.996353    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-credential-socket\") pod \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") "
Nov 01 04:11:34.996604 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.996449    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-kube-api-access-4sgnj\" (UniqueName: \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-kube-api-access-4sgnj\") pod \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") "
Nov 01 04:11:34.996604 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.996557    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-socket" pod "6b33ad10-74bb-4488-bda7-a04a1df5cc8d" (UID: "6b33ad10-74bb-4488-bda7-a04a1df5cc8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:11:34.996604 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.996591    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-envoy\") pod \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") "
Nov 01 04:11:34.996777 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.996637    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-podinfo\") pod \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") "
Nov 01 04:11:34.996777 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.996652    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-credential-socket" pod "6b33ad10-74bb-4488-bda7-a04a1df5cc8d" (UID: "6b33ad10-74bb-4488-bda7-a04a1df5cc8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:11:34.996777 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.996725    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istiod-ca-cert\") pod \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") "
Nov 01 04:11:34.996777 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.996757    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-data\") pod \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") "
Nov 01 04:11:34.996986 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.996807    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-certs\") pod \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\" (UID: \"6b33ad10-74bb-4488-bda7-a04a1df5cc8d\") "
Nov 01 04:11:34.997209 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.997188    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-data" pod "6b33ad10-74bb-4488-bda7-a04a1df5cc8d" (UID: "6b33ad10-74bb-4488-bda7-a04a1df5cc8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:11:34.997287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.997209    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istiod-ca-cert" pod "6b33ad10-74bb-4488-bda7-a04a1df5cc8d" (UID: "6b33ad10-74bb-4488-bda7-a04a1df5cc8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:11:34.997401 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.997378    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-certs" pod "6b33ad10-74bb-4488-bda7-a04a1df5cc8d" (UID: "6b33ad10-74bb-4488-bda7-a04a1df5cc8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:11:34.997626 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.997597    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:34.997626 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.997621    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:34.997762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.997636    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:34.997762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.997649    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:34.997762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.997662    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:34.999218 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.999180    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-envoy" pod "6b33ad10-74bb-4488-bda7-a04a1df5cc8d" (UID: "6b33ad10-74bb-4488-bda7-a04a1df5cc8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:11:34.999218 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.999191    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-token" pod "6b33ad10-74bb-4488-bda7-a04a1df5cc8d" (UID: "6b33ad10-74bb-4488-bda7-a04a1df5cc8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:11:34.999383 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.999280    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-podinfo" pod "6b33ad10-74bb-4488-bda7-a04a1df5cc8d" (UID: "6b33ad10-74bb-4488-bda7-a04a1df5cc8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:11:34.999518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:34.999496    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-kube-api-access-4sgnj" pod "6b33ad10-74bb-4488-bda7-a04a1df5cc8d" (UID: "6b33ad10-74bb-4488-bda7-a04a1df5cc8d"). InnerVolumeSpecName "kube-api-access-4sgnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:11:35.099250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.099150    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:35.099250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.099197    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sgnj\" (UniqueName: \"kubernetes.io/projected/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-kube-api-access-4sgnj\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:35.099250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.099216    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:35.099250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.099230    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b33ad10-74bb-4488-bda7-a04a1df5cc8d-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:35.260957 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.260915    4882 generic.go:356] "Generic (PLEG): container finished" podID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerID="8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b" exitCode=0
Nov 01 04:11:35.261186 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.260971    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"
Nov 01 04:11:35.261186 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.261003    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" event={"ID":"6b33ad10-74bb-4488-bda7-a04a1df5cc8d","Type":"ContainerDied","Data":"8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b"}
Nov 01 04:11:35.261186 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.261093    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6" event={"ID":"6b33ad10-74bb-4488-bda7-a04a1df5cc8d","Type":"ContainerDied","Data":"7235d3fc3999bda56ec0b1174d8b0920093d5aab609f4e272edb6e21c4cf95e3"}
Nov 01 04:11:35.261186 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.261118    4882 scope.go:122] "RemoveContainer" containerID="8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b"
Nov 01 04:11:35.274323 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.274299    4882 scope.go:122] "RemoveContainer" containerID="8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b"
Nov 01 04:11:35.274727 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:11:35.274701    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b\": container with ID starting with 8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b not found: ID does not exist" containerID="8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b"
Nov 01 04:11:35.274842 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.274734    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b"} err="failed to get container status \"8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b\": rpc error: code = NotFound desc = could not find container \"8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b\": container with ID starting with 8c407463bfc7b4ad2cff9ba684d3dbe44531812b530a8028c397ae96d50be68b not found: ID does not exist"
Nov 01 04:11:35.426860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.426771    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"]
Nov 01 04:11:35.432588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.432559    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-jxfk6"]
Nov 01 04:11:35.604817 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:35.604780    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" path="/var/lib/kubelet/pods/6b33ad10-74bb-4488-bda7-a04a1df5cc8d/volumes"
Nov 01 04:11:39.910173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:39.910127    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"]
Nov 01 04:11:39.918100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:39.918071    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6dd36b-8595k"]
Nov 01 04:11:41.600719 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.600685    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="decbaecd-a490-4d3e-bb88-75188792bb92" path="/var/lib/kubelet/pods/decbaecd-a490-4d3e-bb88-75188792bb92/volumes"
Nov 01 04:11:41.796421 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.796379    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"]
Nov 01 04:11:41.797105 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.797082    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy"
Nov 01 04:11:41.797105 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.797106    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy"
Nov 01 04:11:41.797284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.797119    4882 cpu_manager.go:414] "containerMap: removing container" podUID="62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" containerName="istio-proxy"
Nov 01 04:11:41.797284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.797125    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" containerName="istio-proxy"
Nov 01 04:11:41.797284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.797143    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f1421745-fcd8-41fc-ba64-ff1336e43684" containerName="istio-proxy"
Nov 01 04:11:41.797284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.797149    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1421745-fcd8-41fc-ba64-ff1336e43684" containerName="istio-proxy"
Nov 01 04:11:41.797284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.797274    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b33ad10-74bb-4488-bda7-a04a1df5cc8d" containerName="istio-proxy"
Nov 01 04:11:41.797284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.797281    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b4b9a4-ea51-4dd3-93ef-dee4c9cd9c35" containerName="istio-proxy"
Nov 01 04:11:41.797623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.797290    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1421745-fcd8-41fc-ba64-ff1336e43684" containerName="istio-proxy"
Nov 01 04:11:41.802660 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.802637    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:41.851623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.851529    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:11:41.851787 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.851709    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:11:41.854376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.854340    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"]
Nov 01 04:11:41.898544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.898506    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:41.898760 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.898565    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:41.898832 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:41.898761    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkdql\" (UniqueName: \"kubernetes.io/projected/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-kube-api-access-tkdql\") pod \"maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:42.000545 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:42.000504    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:42.000738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:42.000570    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:42.000738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:42.000686    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkdql\" (UniqueName: \"kubernetes.io/projected/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-kube-api-access-tkdql\") pod \"maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:42.001482 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:42.001450    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:42.003152 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:42.003124    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:42.057729 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:42.057697    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkdql\" (UniqueName: \"kubernetes.io/projected/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-kube-api-access-tkdql\") pod \"maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:42.116943 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:42.116848    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:42.491915 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:42.491885    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"]
Nov 01 04:11:42.493783 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:11:42.493742    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdb49cd_fa0e_4779_9fb8_33a2e6a1ec07.slice/crio-fd6ae285fd161e4e5b94ab68b6a9f99805bb60685bd525a9b1fb65147cb2ecd9 WatchSource:0}: Error finding container fd6ae285fd161e4e5b94ab68b6a9f99805bb60685bd525a9b1fb65147cb2ecd9: Status 404 returned error can't find the container with id fd6ae285fd161e4e5b94ab68b6a9f99805bb60685bd525a9b1fb65147cb2ecd9
Nov 01 04:11:43.327160 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:43.327123    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4" event={"ID":"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07","Type":"ContainerStarted","Data":"5eac3fea8fa0c52e0eea0bb473f79101baecf60590c07bce8208c45f2a61a3da"}
Nov 01 04:11:43.327693 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:43.327183    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4" event={"ID":"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07","Type":"ContainerStarted","Data":"fd6ae285fd161e4e5b94ab68b6a9f99805bb60685bd525a9b1fb65147cb2ecd9"}
Nov 01 04:11:43.437081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:43.437028    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4" podStartSLOduration=2.43701051 podStartE2EDuration="2.43701051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:43.435049909 +0000 UTC m=+1771.997082927" watchObservedRunningTime="2026-11-01 04:11:43.43701051 +0000 UTC m=+1771.999043569"
Nov 01 04:11:44.336247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:44.336213    4882 generic.go:356] "Generic (PLEG): container finished" podID="2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07" containerID="5eac3fea8fa0c52e0eea0bb473f79101baecf60590c07bce8208c45f2a61a3da" exitCode=0
Nov 01 04:11:44.336690 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:44.336298    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4" event={"ID":"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07","Type":"ContainerDied","Data":"5eac3fea8fa0c52e0eea0bb473f79101baecf60590c07bce8208c45f2a61a3da"}
Nov 01 04:11:45.499314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:45.499287    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:45.648220 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:45.648108    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-kube-api-access-tkdql\" (UniqueName: \"kubernetes.io/projected/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-kube-api-access-tkdql\") pod \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") "
Nov 01 04:11:45.648220 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:45.648217    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-xks-ca\") pod \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") "
Nov 01 04:11:45.648495 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:45.648259    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-openshift-ca\") pod \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\" (UID: \"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07\") "
Nov 01 04:11:45.648755 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:45.648713    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-openshift-ca" pod "2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07" (UID: "2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:11:45.649679 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:45.649648    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:45.650952 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:45.650918    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-xks-ca" pod "2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07" (UID: "2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:11:45.651070 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:45.651051    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-kube-api-access-tkdql" pod "2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07" (UID: "2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07"). InnerVolumeSpecName "kube-api-access-tkdql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:11:45.750894 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:45.750855    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:45.750894 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:45.750893    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkdql\" (UniqueName: \"kubernetes.io/projected/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07-kube-api-access-tkdql\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:11:46.354851 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:46.354820    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"
Nov 01 04:11:46.355055 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:46.354813    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4" event={"ID":"2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07","Type":"ContainerDied","Data":"fd6ae285fd161e4e5b94ab68b6a9f99805bb60685bd525a9b1fb65147cb2ecd9"}
Nov 01 04:11:46.355055 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:11:46.354960    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd6ae285fd161e4e5b94ab68b6a9f99805bb60685bd525a9b1fb65147cb2ecd9"
Nov 01 04:11:58.725421 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:11:58.725382    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:12:10.820023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:10.819986    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"]
Nov 01 04:12:10.820607 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:10.820213    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" podUID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" containerName="istio-proxy" containerID="cri-o://5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29" gracePeriod=30
Nov 01 04:12:10.834967 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:10.834933    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"]
Nov 01 04:12:10.835171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:10.835149    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk" podUID="03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" containerName="istio-proxy" containerID="cri-o://4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2" gracePeriod=30
Nov 01 04:12:10.838134 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:10.838107    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"]
Nov 01 04:12:10.838387 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:10.838359    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy" containerID="cri-o://f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba" gracePeriod=30
Nov 01 04:12:12.924158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:12.924119    4882 scope.go:122] "RemoveContainer" containerID="be44e9a97a0e01d2d1b560cb3593fa2b44ab5c37a4d37ad087e5baa358e7c14b"
Nov 01 04:12:12.943304 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:12.943278    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:12:12.945259 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:12.945245    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:12:16.144332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.143946    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:12:16.210962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.210931    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:12:16.215371 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.215345    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:12:16.299524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.299480    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-socket\") pod \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") "
Nov 01 04:12:16.299743 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.299547    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-kube-api-access-g5xvf\" (UniqueName: \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-kube-api-access-g5xvf\") pod \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") "
Nov 01 04:12:16.299743 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.299591    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istiod-ca-cert\") pod \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") "
Nov 01 04:12:16.299743 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.299699    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-credential-socket\") pod \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") "
Nov 01 04:12:16.299908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.299765    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-envoy\") pod \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") "
Nov 01 04:12:16.299962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.299911    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-token\" (UniqueName: \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-token\") pod \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") "
Nov 01 04:12:16.299962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.299944    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-podinfo\") pod \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") "
Nov 01 04:12:16.300065 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.299977    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-certs\") pod \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") "
Nov 01 04:12:16.300065 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.300011    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-credential-socket" pod "e426a233-79fe-4e5e-b282-7e1c4cfe7058" (UID: "e426a233-79fe-4e5e-b282-7e1c4cfe7058"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.300065 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.300048    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-data\") pod \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\" (UID: \"e426a233-79fe-4e5e-b282-7e1c4cfe7058\") "
Nov 01 04:12:16.300217 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.300052    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istiod-ca-cert" pod "e426a233-79fe-4e5e-b282-7e1c4cfe7058" (UID: "e426a233-79fe-4e5e-b282-7e1c4cfe7058"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:12:16.300273 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.300247    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-socket" pod "e426a233-79fe-4e5e-b282-7e1c4cfe7058" (UID: "e426a233-79fe-4e5e-b282-7e1c4cfe7058"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.300403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.300344    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-certs" pod "e426a233-79fe-4e5e-b282-7e1c4cfe7058" (UID: "e426a233-79fe-4e5e-b282-7e1c4cfe7058"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.300510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.300471    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-data" pod "e426a233-79fe-4e5e-b282-7e1c4cfe7058" (UID: "e426a233-79fe-4e5e-b282-7e1c4cfe7058"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.300847 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.300751    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.300847 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.300777    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.300847 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.300793    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.300847 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.300807    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.300847 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.300823    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.302362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.302192    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-token" pod "e426a233-79fe-4e5e-b282-7e1c4cfe7058" (UID: "e426a233-79fe-4e5e-b282-7e1c4cfe7058"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:12:16.302517 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.302391    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-envoy" pod "e426a233-79fe-4e5e-b282-7e1c4cfe7058" (UID: "e426a233-79fe-4e5e-b282-7e1c4cfe7058"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.302733 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.302706    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-podinfo" pod "e426a233-79fe-4e5e-b282-7e1c4cfe7058" (UID: "e426a233-79fe-4e5e-b282-7e1c4cfe7058"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:12:16.302733 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.302714    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-kube-api-access-g5xvf" pod "e426a233-79fe-4e5e-b282-7e1c4cfe7058" (UID: "e426a233-79fe-4e5e-b282-7e1c4cfe7058"). InnerVolumeSpecName "kube-api-access-g5xvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:12:16.402516 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.402089    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-kube-api-access-jt8ps\" (UniqueName: \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-kube-api-access-jt8ps\") pod \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") "
Nov 01 04:12:16.402516 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.402156    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-envoy\") pod \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") "
Nov 01 04:12:16.402516 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.402214    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istiod-ca-cert\") pod \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") "
Nov 01 04:12:16.402516 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.402306    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-podinfo\") pod \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") "
Nov 01 04:12:16.402516 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.402350    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-credential-socket\") pod \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") "
Nov 01 04:12:16.402516 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.402386    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-socket\") pod \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") "
Nov 01 04:12:16.404573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.403991    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-socket" pod "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" (UID: "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.404573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.404529    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-credential-socket" pod "e41c7b7c-4552-44f1-b3dd-eb95fe66c466" (UID: "e41c7b7c-4552-44f1-b3dd-eb95fe66c466"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.405099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.405057    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-credential-socket" pod "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" (UID: "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.405691    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-credential-socket\") pod \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") "
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.405762    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-token\" (UniqueName: \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-token\") pod \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") "
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.405806    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-certs\") pod \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") "
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.405882    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-envoy\") pod \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") "
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.405927    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-socket\") pod \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") "
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.405969    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-kube-api-access-vv7cp\" (UniqueName: \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-kube-api-access-vv7cp\") pod \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") "
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.406026    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-data\") pod \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") "
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.406061    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-data\") pod \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\" (UID: \"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5\") "
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.406109    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istiod-ca-cert\") pod \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") "
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.406136    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-certs\") pod \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") "
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.406217    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-podinfo\") pod \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") "
Nov 01 04:12:16.406456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.406287    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-token\" (UniqueName: \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-token\") pod \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\" (UID: \"e41c7b7c-4552-44f1-b3dd-eb95fe66c466\") "
Nov 01 04:12:16.429830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.418222    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istiod-ca-cert" pod "e41c7b7c-4552-44f1-b3dd-eb95fe66c466" (UID: "e41c7b7c-4552-44f1-b3dd-eb95fe66c466"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:12:16.429830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.418520    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-data" pod "e41c7b7c-4552-44f1-b3dd-eb95fe66c466" (UID: "e41c7b7c-4552-44f1-b3dd-eb95fe66c466"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.429830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.418718    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-data" pod "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" (UID: "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.429830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.418902    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-certs" pod "e41c7b7c-4552-44f1-b3dd-eb95fe66c466" (UID: "e41c7b7c-4552-44f1-b3dd-eb95fe66c466"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.429830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.419247    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-certs" pod "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" (UID: "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.429830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.428709    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-token" pod "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" (UID: "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:12:16.429830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.428709    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-socket" pod "e41c7b7c-4552-44f1-b3dd-eb95fe66c466" (UID: "e41c7b7c-4552-44f1-b3dd-eb95fe66c466"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.430500    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istiod-ca-cert" pod "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" (UID: "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431009    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431034    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431051    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431066    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431080    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431094    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431109    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431122    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e426a233-79fe-4e5e-b282-7e1c4cfe7058-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431137    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431152    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431167    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431183    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431196    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431211    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.431226    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5xvf\" (UniqueName: \"kubernetes.io/projected/e426a233-79fe-4e5e-b282-7e1c4cfe7058-kube-api-access-g5xvf\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.432913    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-kube-api-access-vv7cp" pod "e41c7b7c-4552-44f1-b3dd-eb95fe66c466" (UID: "e41c7b7c-4552-44f1-b3dd-eb95fe66c466"). InnerVolumeSpecName "kube-api-access-vv7cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.433225    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-envoy" pod "e41c7b7c-4552-44f1-b3dd-eb95fe66c466" (UID: "e41c7b7c-4552-44f1-b3dd-eb95fe66c466"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.433334    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-podinfo" pod "e41c7b7c-4552-44f1-b3dd-eb95fe66c466" (UID: "e41c7b7c-4552-44f1-b3dd-eb95fe66c466"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.433347    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-kube-api-access-jt8ps" pod "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" (UID: "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5"). InnerVolumeSpecName "kube-api-access-jt8ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:12:16.433824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.433415    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-podinfo" pod "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" (UID: "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:12:16.435616 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.435589    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-token" pod "e41c7b7c-4552-44f1-b3dd-eb95fe66c466" (UID: "e41c7b7c-4552-44f1-b3dd-eb95fe66c466"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:12:16.438284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.438261    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-envoy" pod "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" (UID: "03a0c5ff-b00b-4ca8-9d58-50c4610e23d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:16.532790 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.532753    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.532790 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.532786    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv7cp\" (UniqueName: \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-kube-api-access-vv7cp\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.532790 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.532796    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.533112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.532806    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.533112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.532820    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt8ps\" (UniqueName: \"kubernetes.io/projected/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-kube-api-access-jt8ps\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.533112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.532833    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41c7b7c-4552-44f1-b3dd-eb95fe66c466-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.533112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.532841    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:16.690112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.690071    4882 generic.go:356] "Generic (PLEG): container finished" podID="03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" containerID="4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2" exitCode=0
Nov 01 04:12:16.690112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.690117    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"
Nov 01 04:12:16.690363 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.690153    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk" event={"ID":"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5","Type":"ContainerDied","Data":"4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2"}
Nov 01 04:12:16.690363 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.690207    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk" event={"ID":"03a0c5ff-b00b-4ca8-9d58-50c4610e23d5","Type":"ContainerDied","Data":"d4242ebd9fb15294ab9c646eb6dd0c0677bba4d7a2716375b1d73445f6eb1a31"}
Nov 01 04:12:16.690363 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.690231    4882 scope.go:122] "RemoveContainer" containerID="4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2"
Nov 01 04:12:16.692384 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.692353    4882 generic.go:356] "Generic (PLEG): container finished" podID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerID="f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba" exitCode=0
Nov 01 04:12:16.692579 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.692406    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"
Nov 01 04:12:16.692579 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.692478    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" event={"ID":"e41c7b7c-4552-44f1-b3dd-eb95fe66c466","Type":"ContainerDied","Data":"f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba"}
Nov 01 04:12:16.692579 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.692545    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c" event={"ID":"e41c7b7c-4552-44f1-b3dd-eb95fe66c466","Type":"ContainerDied","Data":"c0df1e4691e56e5c8302e11ffa9d4681e079031bd59f04c9b575028710cf8a68"}
Nov 01 04:12:16.694317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.694292    4882 generic.go:356] "Generic (PLEG): container finished" podID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" containerID="5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29" exitCode=0
Nov 01 04:12:16.694477 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.694346    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"
Nov 01 04:12:16.694477 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.694373    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" event={"ID":"e426a233-79fe-4e5e-b282-7e1c4cfe7058","Type":"ContainerDied","Data":"5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29"}
Nov 01 04:12:16.694477 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.694448    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz" event={"ID":"e426a233-79fe-4e5e-b282-7e1c4cfe7058","Type":"ContainerDied","Data":"fd1c9b4ba914eb186e58d455633bf830e4f74ec24bf210fede001eb5fec53d33"}
Nov 01 04:12:16.703398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.703371    4882 scope.go:122] "RemoveContainer" containerID="4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2"
Nov 01 04:12:16.703817 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:12:16.703778    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2\": container with ID starting with 4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2 not found: ID does not exist" containerID="4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2"
Nov 01 04:12:16.703945 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.703818    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2"} err="failed to get container status \"4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2\": rpc error: code = NotFound desc = could not find container \"4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2\": container with ID starting with 4524afa2968fa44db341c87a20e6e0a7fbb1116f77d1f2587c208b2c099d96d2 not found: ID does not exist"
Nov 01 04:12:16.703945 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.703841    4882 scope.go:122] "RemoveContainer" containerID="f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba"
Nov 01 04:12:16.717268 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.717246    4882 scope.go:122] "RemoveContainer" containerID="ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a"
Nov 01 04:12:16.727719 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.727696    4882 scope.go:122] "RemoveContainer" containerID="f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba"
Nov 01 04:12:16.727993 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:12:16.727974    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba\": container with ID starting with f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba not found: ID does not exist" containerID="f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba"
Nov 01 04:12:16.728046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.728002    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba"} err="failed to get container status \"f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba\": rpc error: code = NotFound desc = could not find container \"f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba\": container with ID starting with f8c329de500057bdc7f204417481c73389985125f37a104986a54231fa486aba not found: ID does not exist"
Nov 01 04:12:16.728046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.728029    4882 scope.go:122] "RemoveContainer" containerID="ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a"
Nov 01 04:12:16.728300 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:12:16.728284    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a\": container with ID starting with ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a not found: ID does not exist" containerID="ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a"
Nov 01 04:12:16.728341 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.728304    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a"} err="failed to get container status \"ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a\": rpc error: code = NotFound desc = could not find container \"ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a\": container with ID starting with ce1ff9b8c16813fefcbddb634e373fc3a3078fdb4e11ead2ebafbc2681580f3a not found: ID does not exist"
Nov 01 04:12:16.728341 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.728317    4882 scope.go:122] "RemoveContainer" containerID="5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29"
Nov 01 04:12:16.738868 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.738839    4882 scope.go:122] "RemoveContainer" containerID="5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29"
Nov 01 04:12:16.739201 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:12:16.739179    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29\": container with ID starting with 5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29 not found: ID does not exist" containerID="5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29"
Nov 01 04:12:16.739281 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.739210    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29"} err="failed to get container status \"5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29\": rpc error: code = NotFound desc = could not find container \"5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29\": container with ID starting with 5c425a9374ba4ff56bae53c9ef9b05f426846f42056b1d14d1c8518aca074c29 not found: ID does not exist"
Nov 01 04:12:16.846131 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.846091    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"]
Nov 01 04:12:16.850796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.850730    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-xvckz"]
Nov 01 04:12:16.952249 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.952158    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"]
Nov 01 04:12:16.959763 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:16.959727    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-gdwpk"]
Nov 01 04:12:17.064649 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:17.064611    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"]
Nov 01 04:12:17.069015 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:17.068981    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-5vq8c"]
Nov 01 04:12:17.599780 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:17.599746    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" path="/var/lib/kubelet/pods/03a0c5ff-b00b-4ca8-9d58-50c4610e23d5/volumes"
Nov 01 04:12:17.600293 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:17.600277    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" path="/var/lib/kubelet/pods/e41c7b7c-4552-44f1-b3dd-eb95fe66c466/volumes"
Nov 01 04:12:17.600772 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:17.600756    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" path="/var/lib/kubelet/pods/e426a233-79fe-4e5e-b282-7e1c4cfe7058/volumes"
Nov 01 04:12:18.049290 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:18.049257    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"]
Nov 01 04:12:18.095268 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:18.095224    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0d26b1d-9vmt4"]
Nov 01 04:12:19.610374 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.610332    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8958cd1-dbed-4f7a-8098-94a80bec26a5" path="/var/lib/kubelet/pods/b8958cd1-dbed-4f7a-8098-94a80bec26a5/volumes"
Nov 01 04:12:19.926116 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926031    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"]
Nov 01 04:12:19.926591 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926576    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy"
Nov 01 04:12:19.926648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926592    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy"
Nov 01 04:12:19.926648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926607    4882 cpu_manager.go:414] "containerMap: removing container" podUID="03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" containerName="istio-proxy"
Nov 01 04:12:19.926648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926612    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" containerName="istio-proxy"
Nov 01 04:12:19.926648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926622    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" containerName="istio-proxy"
Nov 01 04:12:19.926648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926628    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" containerName="istio-proxy"
Nov 01 04:12:19.926648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926642    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07" containerName="revoke-keys"
Nov 01 04:12:19.926648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926647    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07" containerName="revoke-keys"
Nov 01 04:12:19.926884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926655    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy"
Nov 01 04:12:19.926884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926661    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy"
Nov 01 04:12:19.926884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926744    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07" containerName="revoke-keys"
Nov 01 04:12:19.926884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926754    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy"
Nov 01 04:12:19.926884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926767    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a0c5ff-b00b-4ca8-9d58-50c4610e23d5" containerName="istio-proxy"
Nov 01 04:12:19.926884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.926773    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e426a233-79fe-4e5e-b282-7e1c4cfe7058" containerName="istio-proxy"
Nov 01 04:12:19.931872 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:19.931849    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:20.001139 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.001094    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc08a306-q7wcp\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:20.001321 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.001291    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrvcb\" (UniqueName: \"kubernetes.io/projected/60799821-4071-47ef-b9dc-86189e2585cd-kube-api-access-mrvcb\") pod \"maas-api-revoke-sub-cc08a306-q7wcp\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:20.001478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.001461    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc08a306-q7wcp\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:20.026373 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.026338    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:12:20.026588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.026534    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:12:20.031745 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.031713    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"]
Nov 01 04:12:20.102739 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.102700    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc08a306-q7wcp\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:20.102954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.102866    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc08a306-q7wcp\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:20.103016 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.102956    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrvcb\" (UniqueName: \"kubernetes.io/projected/60799821-4071-47ef-b9dc-86189e2585cd-kube-api-access-mrvcb\") pod \"maas-api-revoke-sub-cc08a306-q7wcp\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:20.105612 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.105579    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc08a306-q7wcp\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:20.106870 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.106671    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc08a306-q7wcp\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:20.161102 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.161059    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrvcb\" (UniqueName: \"kubernetes.io/projected/60799821-4071-47ef-b9dc-86189e2585cd-kube-api-access-mrvcb\") pod \"maas-api-revoke-sub-cc08a306-q7wcp\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:20.246156 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.246120    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:20.620522 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.620486    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"]
Nov 01 04:12:20.622263 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:12:20.622233    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60799821_4071_47ef_b9dc_86189e2585cd.slice/crio-2e64c325d3b3a4b66e8039e6cfe6c2e9e0a216857c19bf2587ac5789021929f7 WatchSource:0}: Error finding container 2e64c325d3b3a4b66e8039e6cfe6c2e9e0a216857c19bf2587ac5789021929f7: Status 404 returned error can't find the container with id 2e64c325d3b3a4b66e8039e6cfe6c2e9e0a216857c19bf2587ac5789021929f7
Nov 01 04:12:20.734318 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:20.734279    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp" event={"ID":"60799821-4071-47ef-b9dc-86189e2585cd","Type":"ContainerStarted","Data":"2e64c325d3b3a4b66e8039e6cfe6c2e9e0a216857c19bf2587ac5789021929f7"}
Nov 01 04:12:21.742485 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:21.742384    4882 generic.go:356] "Generic (PLEG): container finished" podID="60799821-4071-47ef-b9dc-86189e2585cd" containerID="50ff95a5b8151488c39f38f9a8f494b5dfa99752a053ac60d7eb702ff64f4c63" exitCode=0
Nov 01 04:12:21.742485 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:21.742467    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp" event={"ID":"60799821-4071-47ef-b9dc-86189e2585cd","Type":"ContainerDied","Data":"50ff95a5b8151488c39f38f9a8f494b5dfa99752a053ac60d7eb702ff64f4c63"}
Nov 01 04:12:22.925545 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:22.925519    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:23.042531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.042440    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-openshift-ca\") pod \"60799821-4071-47ef-b9dc-86189e2585cd\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") "
Nov 01 04:12:23.042716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.042541    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60799821-4071-47ef-b9dc-86189e2585cd-kube-api-access-mrvcb\" (UniqueName: \"kubernetes.io/projected/60799821-4071-47ef-b9dc-86189e2585cd-kube-api-access-mrvcb\") pod \"60799821-4071-47ef-b9dc-86189e2585cd\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") "
Nov 01 04:12:23.042716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.042592    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-xks-ca\") pod \"60799821-4071-47ef-b9dc-86189e2585cd\" (UID: \"60799821-4071-47ef-b9dc-86189e2585cd\") "
Nov 01 04:12:23.042839 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.042735    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-openshift-ca" pod "60799821-4071-47ef-b9dc-86189e2585cd" (UID: "60799821-4071-47ef-b9dc-86189e2585cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:12:23.043143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.043122    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:23.045037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.045008    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-xks-ca" pod "60799821-4071-47ef-b9dc-86189e2585cd" (UID: "60799821-4071-47ef-b9dc-86189e2585cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:12:23.045387 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.045364    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60799821-4071-47ef-b9dc-86189e2585cd-kube-api-access-mrvcb" pod "60799821-4071-47ef-b9dc-86189e2585cd" (UID: "60799821-4071-47ef-b9dc-86189e2585cd"). InnerVolumeSpecName "kube-api-access-mrvcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:12:23.144842 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.144804    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrvcb\" (UniqueName: \"kubernetes.io/projected/60799821-4071-47ef-b9dc-86189e2585cd-kube-api-access-mrvcb\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:23.144842 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.144836    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60799821-4071-47ef-b9dc-86189e2585cd-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:12:23.759614 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.759582    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"
Nov 01 04:12:23.759788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.759620    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp" event={"ID":"60799821-4071-47ef-b9dc-86189e2585cd","Type":"ContainerDied","Data":"2e64c325d3b3a4b66e8039e6cfe6c2e9e0a216857c19bf2587ac5789021929f7"}
Nov 01 04:12:23.759788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:23.759670    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e64c325d3b3a4b66e8039e6cfe6c2e9e0a216857c19bf2587ac5789021929f7"
Nov 01 04:12:28.725159 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:12:28.725120    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:12:46.008104 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:46.008057    4882 generic.go:356] "Generic (PLEG): container finished" podID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerID="cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01" exitCode=0
Nov 01 04:12:46.008684 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:46.008129    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" event={"ID":"865370fa-7d36-46e3-98e2-88ca8d35c6b5","Type":"ContainerDied","Data":"cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01"}
Nov 01 04:12:46.008750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:46.008731    4882 scope.go:122] "RemoveContainer" containerID="cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01"
Nov 01 04:12:46.010346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:46.010316    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:12:46.010485 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:46.010386    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:12:46.010485 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:46.010418    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:12:47.019056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:47.019017    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" event={"ID":"865370fa-7d36-46e3-98e2-88ca8d35c6b5","Type":"ContainerStarted","Data":"b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053"}
Nov 01 04:12:47.072017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:47.071980    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:12:47.086655 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:47.086608    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 01 04:12:47.086872 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:47.086697    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 01 04:12:48.081454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:48.079839    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 01 04:12:48.081454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:48.079920    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 01 04:12:49.081453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:49.072796    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 01 04:12:49.081453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:49.072859    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 01 04:12:50.078454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:50.074800    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 01 04:12:50.078454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:50.074868    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 01 04:12:51.086455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:51.085461    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 01 04:12:51.086455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:51.085526    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 01 04:12:52.074594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:52.074553    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 01 04:12:52.074925 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:52.074884    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 01 04:12:53.069198 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:53.069161    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:12:53.069618 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:53.069552    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:12:53.070340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:12:53.070320    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:12:58.725185 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:12:58.725146    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:13:00.361119 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.361086    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"]
Nov 01 04:13:00.361682 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.361666    4882 cpu_manager.go:414] "containerMap: removing container" podUID="60799821-4071-47ef-b9dc-86189e2585cd" containerName="revoke-keys"
Nov 01 04:13:00.361731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.361683    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60799821-4071-47ef-b9dc-86189e2585cd" containerName="revoke-keys"
Nov 01 04:13:00.361820 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.361810    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="60799821-4071-47ef-b9dc-86189e2585cd" containerName="revoke-keys"
Nov 01 04:13:00.361918 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.361821    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41c7b7c-4552-44f1-b3dd-eb95fe66c466" containerName="istio-proxy"
Nov 01 04:13:00.365712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.365681    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.466970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.466920    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.466970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.466973    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fftkv\" (UniqueName: \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-kube-api-access-fftkv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.467256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.467108    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.467256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.467155    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.467375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.467273    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.467375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.467362    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.467515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.467406    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.467515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.467472    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fbfedcc-725a-4408-883b-62a2c49e655e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.467515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.467488    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.478008 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.477969    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"]
Nov 01 04:13:00.568706 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.568670    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.568923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.568901    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.568998 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.568983    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.569058 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.569034    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fbfedcc-725a-4408-883b-62a2c49e655e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.569267 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.569066    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.569267 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.569140    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.569267 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.569168    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fftkv\" (UniqueName: \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-kube-api-access-fftkv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.569267 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.569140    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.569501 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.569284    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.569501 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.569345    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.569615 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.569492    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.569751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.569724    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.569839 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.569819    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fbfedcc-725a-4408-883b-62a2c49e655e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.569998 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.569984    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.571279 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.571254    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.571901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.571879    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.626625 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.626545    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.626762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.626681    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fftkv\" (UniqueName: \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-kube-api-access-fftkv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-plhn5\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:00.679653 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:00.679618    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:01.104961 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:01.104933    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"]
Nov 01 04:13:01.107501 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:13:01.107469    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbfedcc_725a_4408_883b_62a2c49e655e.slice/crio-8f2ee0c3a0f08dcb66276769ee284e6076db9ebd15be683a7dce12507884e0c3 WatchSource:0}: Error finding container 8f2ee0c3a0f08dcb66276769ee284e6076db9ebd15be683a7dce12507884e0c3: Status 404 returned error can't find the container with id 8f2ee0c3a0f08dcb66276769ee284e6076db9ebd15be683a7dce12507884e0c3
Nov 01 04:13:01.110226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:01.110187    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:13:01.110347 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:01.110274    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:13:01.110347 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:01.110321    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:13:02.124299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:02.124265    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" event={"ID":"1fbfedcc-725a-4408-883b-62a2c49e655e","Type":"ContainerStarted","Data":"21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2"}
Nov 01 04:13:02.124722 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:02.124323    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" event={"ID":"1fbfedcc-725a-4408-883b-62a2c49e655e","Type":"ContainerStarted","Data":"8f2ee0c3a0f08dcb66276769ee284e6076db9ebd15be683a7dce12507884e0c3"}
Nov 01 04:13:02.304926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:02.304864    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" podStartSLOduration=2.304847646 podStartE2EDuration="2.304847646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:13:02.300599781 +0000 UTC m=+1850.862632803" watchObservedRunningTime="2026-11-01 04:13:02.304847646 +0000 UTC m=+1850.866880662"
Nov 01 04:13:02.680892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:02.680849    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:02.683728 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:02.683695    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-plhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 04:13:02.683979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:02.683949    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 04:13:03.683522 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:03.681906    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-plhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 04:13:03.683522 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:03.681978    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 04:13:04.686891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:04.686848    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-plhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 04:13:04.687340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:04.686928    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 04:13:05.683453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:05.682827    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-plhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 04:13:05.683453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:05.682897    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 04:13:06.682453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:06.681954    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-plhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 04:13:06.682453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:06.682066    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 04:13:07.680870 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:07.680834    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-plhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 04:13:07.681059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:07.680904    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 04:13:08.680557 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:08.680521    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-plhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 04:13:08.680997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:08.680579    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 04:13:09.684737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:09.684697    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:09.685160 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:09.685114    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:09.685897 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:09.685868    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:13:12.962131 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:12.962100    4882 scope.go:122] "RemoveContainer" containerID="0366e78c286312e272e9ec3b9b05f02737fb7f54bea198852bd7f6694ae97759"
Nov 01 04:13:12.972636 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:12.972613    4882 scope.go:122] "RemoveContainer" containerID="0b06e6a29ee88e0cb50bf78407c35dd6a10b6232061f262d3b7650f14d06a9f3"
Nov 01 04:13:23.898390 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:23.898353    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"]
Nov 01 04:13:23.944493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:23.944458    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dc980b-k7lc4"]
Nov 01 04:13:25.599846 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:25.599809    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07" path="/var/lib/kubelet/pods/2bdb49cd-fa0e-4779-9fb8-33a2e6a1ec07/volumes"
Nov 01 04:13:28.797667 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:13:28.797625    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:13:30.719833 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:30.719791    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"]
Nov 01 04:13:30.720273 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:30.720011    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy" containerID="cri-o://89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc" gracePeriod=30
Nov 01 04:13:31.641856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:31.641814    4882 patch_prober.go:29] interesting pod/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:13:31.642041 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:31.641885    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:13:34.113573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.113534    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"]
Nov 01 04:13:34.143798 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.143763    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.192646 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.189273    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"]
Nov 01 04:13:34.225192 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.225154    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-131718-openshift-default-dockercfg-r7b4w\""
Nov 01 04:13:34.279244 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.279208    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-envoy\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.279497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.279258    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-data\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.279497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.279399    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-token\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.279497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.279468    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4abcc8-b995-4851-9a00-5514e866ad89-istiod-ca-cert\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.279705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.279503    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-podinfo\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.279705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.279535    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-socket\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.279705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.279581    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-certs\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.279705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.279646    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhtms\" (UniqueName: \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-kube-api-access-qhtms\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.279705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.279694    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-credential-socket\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.381245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.381143    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-token\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.381245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.381211    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4abcc8-b995-4851-9a00-5514e866ad89-istiod-ca-cert\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.381527 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.381255    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-podinfo\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.381527 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.381288    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-socket\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.381527 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.381315    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-certs\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.381527 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.381367    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhtms\" (UniqueName: \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-kube-api-access-qhtms\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.381527 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.381443    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-credential-socket\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.381784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.381631    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-envoy\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.381784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.381675    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-data\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.381894 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.381791    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-certs\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.381957 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.381896    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-credential-socket\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.382091 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.382069    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-data\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.382184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.382091    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-socket\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.382664 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.382629    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4abcc8-b995-4851-9a00-5514e866ad89-istiod-ca-cert\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.384189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.384163    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-envoy\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.384665 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.384642    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-podinfo\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.453415 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.453381    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-token\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.453585 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.453539    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhtms\" (UniqueName: \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-kube-api-access-qhtms\") pod \"e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") " pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.525795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.525754    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:34.950884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.950859    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"]
Nov 01 04:13:34.953050 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:13:34.953018    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4abcc8_b995_4851_9a00_5514e866ad89.slice/crio-6a5d15470ca7a1dc9e3b730c5be1344852c2d46c388e57d60209f892626afb61 WatchSource:0}: Error finding container 6a5d15470ca7a1dc9e3b730c5be1344852c2d46c388e57d60209f892626afb61: Status 404 returned error can't find the container with id 6a5d15470ca7a1dc9e3b730c5be1344852c2d46c388e57d60209f892626afb61
Nov 01 04:13:34.955750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.955713    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:13:34.955849 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.955785    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:13:34.955849 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:34.955816    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:13:35.436205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:35.436163    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" event={"ID":"2c4abcc8-b995-4851-9a00-5514e866ad89","Type":"ContainerStarted","Data":"498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc"}
Nov 01 04:13:35.436673 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:35.436232    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" event={"ID":"2c4abcc8-b995-4851-9a00-5514e866ad89","Type":"ContainerStarted","Data":"6a5d15470ca7a1dc9e3b730c5be1344852c2d46c388e57d60209f892626afb61"}
Nov 01 04:13:35.526113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:35.526075    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:35.550552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:35.550500    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" podStartSLOduration=1.550486313 podStartE2EDuration="1.550486313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:13:35.548735577 +0000 UTC m=+1884.110768594" watchObservedRunningTime="2026-11-01 04:13:35.550486313 +0000 UTC m=+1884.112519329"
Nov 01 04:13:36.045475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.037597    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:13:36.215785 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.215748    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bfa667-16e5-4748-a44e-fa057ace864e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bfa667-16e5-4748-a44e-fa057ace864e-istiod-ca-cert\") pod \"48bfa667-16e5-4748-a44e-fa057ace864e\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") "
Nov 01 04:13:36.215984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.215840    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-data\") pod \"48bfa667-16e5-4748-a44e-fa057ace864e\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") "
Nov 01 04:13:36.215984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.215875    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-kube-api-access-65wff\" (UniqueName: \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-kube-api-access-65wff\") pod \"48bfa667-16e5-4748-a44e-fa057ace864e\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") "
Nov 01 04:13:36.215984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.215897    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-socket\") pod \"48bfa667-16e5-4748-a44e-fa057ace864e\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") "
Nov 01 04:13:36.215984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.215967    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48bfa667-16e5-4748-a44e-fa057ace864e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bfa667-16e5-4748-a44e-fa057ace864e-istio-podinfo\") pod \"48bfa667-16e5-4748-a44e-fa057ace864e\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") "
Nov 01 04:13:36.216212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216000    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-credential-socket\") pod \"48bfa667-16e5-4748-a44e-fa057ace864e\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") "
Nov 01 04:13:36.216212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216032    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-envoy\") pod \"48bfa667-16e5-4748-a44e-fa057ace864e\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") "
Nov 01 04:13:36.216212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216068    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-certs\") pod \"48bfa667-16e5-4748-a44e-fa057ace864e\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") "
Nov 01 04:13:36.216212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216111    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-istio-token\" (UniqueName: \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-istio-token\") pod \"48bfa667-16e5-4748-a44e-fa057ace864e\" (UID: \"48bfa667-16e5-4748-a44e-fa057ace864e\") "
Nov 01 04:13:36.216212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216160    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bfa667-16e5-4748-a44e-fa057ace864e-istiod-ca-cert" pod "48bfa667-16e5-4748-a44e-fa057ace864e" (UID: "48bfa667-16e5-4748-a44e-fa057ace864e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:13:36.216212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216202    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-data" pod "48bfa667-16e5-4748-a44e-fa057ace864e" (UID: "48bfa667-16e5-4748-a44e-fa057ace864e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:36.216593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216375    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-credential-socket" pod "48bfa667-16e5-4748-a44e-fa057ace864e" (UID: "48bfa667-16e5-4748-a44e-fa057ace864e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:36.216593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216331    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-socket" pod "48bfa667-16e5-4748-a44e-fa057ace864e" (UID: "48bfa667-16e5-4748-a44e-fa057ace864e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:36.216593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216550    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-certs" pod "48bfa667-16e5-4748-a44e-fa057ace864e" (UID: "48bfa667-16e5-4748-a44e-fa057ace864e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:36.216861 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216839    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:36.216958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216867    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:36.216958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216884    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bfa667-16e5-4748-a44e-fa057ace864e-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:36.216958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216899    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:36.216958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.216913    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:36.218344 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.218316    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-kube-api-access-65wff" pod "48bfa667-16e5-4748-a44e-fa057ace864e" (UID: "48bfa667-16e5-4748-a44e-fa057ace864e"). InnerVolumeSpecName "kube-api-access-65wff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:13:36.218480 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.218321    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48bfa667-16e5-4748-a44e-fa057ace864e-istio-podinfo" pod "48bfa667-16e5-4748-a44e-fa057ace864e" (UID: "48bfa667-16e5-4748-a44e-fa057ace864e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:13:36.218480 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.218382    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-istio-token" pod "48bfa667-16e5-4748-a44e-fa057ace864e" (UID: "48bfa667-16e5-4748-a44e-fa057ace864e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:13:36.218626 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.218609    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-envoy" pod "48bfa667-16e5-4748-a44e-fa057ace864e" (UID: "48bfa667-16e5-4748-a44e-fa057ace864e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:36.318891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.318795    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65wff\" (UniqueName: \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-kube-api-access-65wff\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:36.318891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.318833    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bfa667-16e5-4748-a44e-fa057ace864e-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:36.318891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.318844    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bfa667-16e5-4748-a44e-fa057ace864e-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:36.318891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.318853    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bfa667-16e5-4748-a44e-fa057ace864e-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:36.444364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.444324    4882 generic.go:356] "Generic (PLEG): container finished" podID="48bfa667-16e5-4748-a44e-fa057ace864e" containerID="89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc" exitCode=0
Nov 01 04:13:36.444848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.444373    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"
Nov 01 04:13:36.444848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.444408    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" event={"ID":"48bfa667-16e5-4748-a44e-fa057ace864e","Type":"ContainerDied","Data":"89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc"}
Nov 01 04:13:36.444848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.444512    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5" event={"ID":"48bfa667-16e5-4748-a44e-fa057ace864e","Type":"ContainerDied","Data":"f78d8aa5203d75462a8013fbaeab5d7d4313225add2de6b9108a1047d28b8964"}
Nov 01 04:13:36.444848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.444540    4882 scope.go:122] "RemoveContainer" containerID="89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc"
Nov 01 04:13:36.455350 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.455332    4882 scope.go:122] "RemoveContainer" containerID="5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba"
Nov 01 04:13:36.467222 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.467205    4882 scope.go:122] "RemoveContainer" containerID="89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc"
Nov 01 04:13:36.467478 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:13:36.467461    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc\": container with ID starting with 89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc not found: ID does not exist" containerID="89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc"
Nov 01 04:13:36.467539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.467487    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc"} err="failed to get container status \"89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc\": rpc error: code = NotFound desc = could not find container \"89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc\": container with ID starting with 89da042b10c495d29908dc6fcc0914c5f273dc458e4821a5a9544620e83c2cbc not found: ID does not exist"
Nov 01 04:13:36.467539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.467505    4882 scope.go:122] "RemoveContainer" containerID="5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba"
Nov 01 04:13:36.467722 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:13:36.467703    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba\": container with ID starting with 5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba not found: ID does not exist" containerID="5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba"
Nov 01 04:13:36.467785 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.467732    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba"} err="failed to get container status \"5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba\": rpc error: code = NotFound desc = could not find container \"5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba\": container with ID starting with 5db321a574c69de21ffac388103234c29ba0962c0b631ca27bbb3f24182ccdba not found: ID does not exist"
Nov 01 04:13:36.532252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.532227    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:36.602795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.602703    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"]
Nov 01 04:13:36.646421 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:36.646387    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dc980b-openshift-default-5bd94f7b59-lxmd5"]
Nov 01 04:13:37.452094 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:37.452059    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:37.453071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:37.453046    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:13:37.600477 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:37.600418    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" path="/var/lib/kubelet/pods/48bfa667-16e5-4748-a44e-fa057ace864e/volumes"
Nov 01 04:13:45.664459 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.664411    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"]
Nov 01 04:13:45.664965 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.664951    4882 cpu_manager.go:414] "containerMap: removing container" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy"
Nov 01 04:13:45.665011 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.664966    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy"
Nov 01 04:13:45.665011 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.664985    4882 cpu_manager.go:414] "containerMap: removing container" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy"
Nov 01 04:13:45.665011 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.664991    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy"
Nov 01 04:13:45.665155 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.665142    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy"
Nov 01 04:13:45.665190 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.665160    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bfa667-16e5-4748-a44e-fa057ace864e" containerName="istio-proxy"
Nov 01 04:13:45.670243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.670217    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.774123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.771865    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"]
Nov 01 04:13:45.812097 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.812050    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4921413-f98d-4883-a532-a4e1c45219d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.812097 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.812099    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.812365 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.812171    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4921413-f98d-4883-a532-a4e1c45219d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.812365 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.812314    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.812365 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.812358    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.812497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.812459    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.812536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.812527    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwk26\" (UniqueName: \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-kube-api-access-mwk26\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.812609 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.812595    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.812648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.812630    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.914364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.914323    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.914364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.914364    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.914696 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.914394    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.914696 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.914458    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwk26\" (UniqueName: \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-kube-api-access-mwk26\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.914696 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.914556    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.914696 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.914602    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.914928 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.914738    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4921413-f98d-4883-a532-a4e1c45219d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.914928 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.914771    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.914928 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.914797    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4921413-f98d-4883-a532-a4e1c45219d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.914928 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.914913    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.915506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.914958    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.915506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.915092    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.915506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.915124    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.915506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.915398    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4921413-f98d-4883-a532-a4e1c45219d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.916751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.916729    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.917224 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.917200    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4921413-f98d-4883-a532-a4e1c45219d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.972208 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.972174    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.972372 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.972183    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwk26\" (UniqueName: \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-kube-api-access-mwk26\") pod \"maas-default-gateway-openshift-default-5df48fcc99-d9tgt\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:45.983691 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:45.983660    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:46.413448 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:46.413397    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"]
Nov 01 04:13:46.415544 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:13:46.415513    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4921413_f98d_4883_a532_a4e1c45219d7.slice/crio-6834917fc4ba04e6d1060c8e557cdae40d1fcac704577b69d234fd27bbec3eb9 WatchSource:0}: Error finding container 6834917fc4ba04e6d1060c8e557cdae40d1fcac704577b69d234fd27bbec3eb9: Status 404 returned error can't find the container with id 6834917fc4ba04e6d1060c8e557cdae40d1fcac704577b69d234fd27bbec3eb9
Nov 01 04:13:46.418236 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:46.418203    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:13:46.418327 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:46.418272    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:13:46.418327 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:46.418303    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:13:46.536236 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:46.536191    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" event={"ID":"a4921413-f98d-4883-a532-a4e1c45219d7","Type":"ContainerStarted","Data":"ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3"}
Nov 01 04:13:46.536475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:46.536265    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" event={"ID":"a4921413-f98d-4883-a532-a4e1c45219d7","Type":"ContainerStarted","Data":"6834917fc4ba04e6d1060c8e557cdae40d1fcac704577b69d234fd27bbec3eb9"}
Nov 01 04:13:46.651887 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:46.651822    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" podStartSLOduration=1.651806587 podStartE2EDuration="1.651806587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:13:46.649014062 +0000 UTC m=+1895.211047081" watchObservedRunningTime="2026-11-01 04:13:46.651806587 +0000 UTC m=+1895.213839605"
Nov 01 04:13:46.993583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:46.989545    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:47.988270 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:47.988234    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-d9tgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 04:13:47.988492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:47.988301    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 04:13:48.985454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:48.984748    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-d9tgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 04:13:48.985454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:48.984813    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 04:13:49.934254 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:49.934219    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"]
Nov 01 04:13:49.934519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:49.934486    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy" containerID="cri-o://b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1" gracePeriod=30
Nov 01 04:13:49.989491 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:49.989454    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-d9tgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 04:13:49.990109 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:49.990056    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 04:13:49.990482 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:49.990416    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"]
Nov 01 04:13:49.990929 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:49.990868    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy" containerID="cri-o://8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a" gracePeriod=30
Nov 01 04:13:50.986857 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:50.986814    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-d9tgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 04:13:50.987048 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:50.986902    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 04:13:51.621392 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:51.621355    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:13:51.621814 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:51.621423    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:13:51.998462 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:51.994285    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-d9tgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 04:13:52.003580 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:52.003509    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 04:13:52.984990 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:52.984949    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-d9tgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 04:13:52.985424 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:52.985018    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 04:13:53.158766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:53.158727    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"]
Nov 01 04:13:53.159034 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:53.159004    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerName="istio-proxy" containerID="cri-o://4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55" gracePeriod=30
Nov 01 04:13:53.207188 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:53.207147    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:13:53.207498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:53.207422    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:13:53.984738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:53.984704    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-d9tgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 04:13:53.985005 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:53.984763    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 04:13:55.311061 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.311026    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:13:55.348014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.347983    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 04:13:55.364293 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.364254    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:13:55.364489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.364320    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:13:55.385410 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385363    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-podinfo\") pod \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") "
Nov 01 04:13:55.385602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385475    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9975f6aa-7802-46fc-9612-fd00544fa547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9975f6aa-7802-46fc-9612-fd00544fa547-istiod-ca-cert\") pod \"9975f6aa-7802-46fc-9612-fd00544fa547\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") "
Nov 01 04:13:55.385602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385522    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-data\") pod \"9975f6aa-7802-46fc-9612-fd00544fa547\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") "
Nov 01 04:13:55.385602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385552    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-envoy\") pod \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") "
Nov 01 04:13:55.385796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385606    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-socket\") pod \"9975f6aa-7802-46fc-9612-fd00544fa547\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") "
Nov 01 04:13:55.385796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385632    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-credential-socket\") pod \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") "
Nov 01 04:13:55.385796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385669    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-certs\") pod \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") "
Nov 01 04:13:55.385796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385718    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-istio-token\" (UniqueName: \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-istio-token\") pod \"9975f6aa-7802-46fc-9612-fd00544fa547\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") "
Nov 01 04:13:55.385796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385770    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istiod-ca-cert\") pod \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") "
Nov 01 04:13:55.386071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385810    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-kube-api-access-9fbm8\" (UniqueName: \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-kube-api-access-9fbm8\") pod \"9975f6aa-7802-46fc-9612-fd00544fa547\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") "
Nov 01 04:13:55.386071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385900    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-socket\") pod \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") "
Nov 01 04:13:55.386071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385915    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9975f6aa-7802-46fc-9612-fd00544fa547-istiod-ca-cert" pod "9975f6aa-7802-46fc-9612-fd00544fa547" (UID: "9975f6aa-7802-46fc-9612-fd00544fa547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:13:55.386071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385952    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-kube-api-access-66hg5\" (UniqueName: \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-kube-api-access-66hg5\") pod \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") "
Nov 01 04:13:55.386071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.385994    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-envoy\") pod \"9975f6aa-7802-46fc-9612-fd00544fa547\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") "
Nov 01 04:13:55.386071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.386023    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9975f6aa-7802-46fc-9612-fd00544fa547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9975f6aa-7802-46fc-9612-fd00544fa547-istio-podinfo\") pod \"9975f6aa-7802-46fc-9612-fd00544fa547\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") "
Nov 01 04:13:55.386071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.386034    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-credential-socket" pod "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" (UID: "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:55.386071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.386054    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-data\") pod \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") "
Nov 01 04:13:55.386518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.386129    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-token\" (UniqueName: \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-token\") pod \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\" (UID: \"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f\") "
Nov 01 04:13:55.386518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.386213    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-credential-socket\") pod \"9975f6aa-7802-46fc-9612-fd00544fa547\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") "
Nov 01 04:13:55.386518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.386246    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-socket" pod "9975f6aa-7802-46fc-9612-fd00544fa547" (UID: "9975f6aa-7802-46fc-9612-fd00544fa547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:55.386518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.386279    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-certs\") pod \"9975f6aa-7802-46fc-9612-fd00544fa547\" (UID: \"9975f6aa-7802-46fc-9612-fd00544fa547\") "
Nov 01 04:13:55.386518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.386406    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-data" pod "9975f6aa-7802-46fc-9612-fd00544fa547" (UID: "9975f6aa-7802-46fc-9612-fd00544fa547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:55.387030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.386991    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9975f6aa-7802-46fc-9612-fd00544fa547-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.387030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.387021    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.387181 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.387040    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.387181 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.387058    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.389049 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.387543    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istiod-ca-cert" pod "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" (UID: "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:13:55.389049 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.387693    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-certs" pod "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" (UID: "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:55.389049 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.387935    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-data" pod "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" (UID: "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:55.389049 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.388174    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-certs" pod "9975f6aa-7802-46fc-9612-fd00544fa547" (UID: "9975f6aa-7802-46fc-9612-fd00544fa547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:55.389049 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.388449    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-credential-socket" pod "9975f6aa-7802-46fc-9612-fd00544fa547" (UID: "9975f6aa-7802-46fc-9612-fd00544fa547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:55.389049 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.388814    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-envoy" pod "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" (UID: "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:55.389436 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.389060    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-socket" pod "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" (UID: "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:55.390367 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.390343    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-kube-api-access-66hg5" pod "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" (UID: "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f"). InnerVolumeSpecName "kube-api-access-66hg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:13:55.390721 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.390699    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-podinfo" pod "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" (UID: "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:13:55.391582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.391556    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-kube-api-access-9fbm8" pod "9975f6aa-7802-46fc-9612-fd00544fa547" (UID: "9975f6aa-7802-46fc-9612-fd00544fa547"). InnerVolumeSpecName "kube-api-access-9fbm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:13:55.391904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.391883    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-envoy" pod "9975f6aa-7802-46fc-9612-fd00544fa547" (UID: "9975f6aa-7802-46fc-9612-fd00544fa547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:55.392608 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.392586    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9975f6aa-7802-46fc-9612-fd00544fa547-istio-podinfo" pod "9975f6aa-7802-46fc-9612-fd00544fa547" (UID: "9975f6aa-7802-46fc-9612-fd00544fa547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:13:55.392693 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.392607    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-token" pod "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" (UID: "dc94dbdc-bb4f-4b4b-aa94-a516adfb041f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:13:55.394502 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.394472    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-istio-token" pod "9975f6aa-7802-46fc-9612-fd00544fa547" (UID: "9975f6aa-7802-46fc-9612-fd00544fa547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:13:55.488832 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488786    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.488832 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488831    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488846    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488860    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488874    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488889    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fbm8\" (UniqueName: \"kubernetes.io/projected/9975f6aa-7802-46fc-9612-fd00544fa547-kube-api-access-9fbm8\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488906    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488920    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66hg5\" (UniqueName: \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-kube-api-access-66hg5\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488935    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488947    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9975f6aa-7802-46fc-9612-fd00544fa547-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488960    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488973    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.488988    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.489123 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.489002    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9975f6aa-7802-46fc-9612-fd00544fa547-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:55.621369 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.621339    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:55.622602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.622581    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:55.626816 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.626791    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:13:55.639411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.639377    4882 generic.go:356] "Generic (PLEG): container finished" podID="9975f6aa-7802-46fc-9612-fd00544fa547" containerID="b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1" exitCode=0
Nov 01 04:13:55.639623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.639460    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"
Nov 01 04:13:55.639623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.639486    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" event={"ID":"9975f6aa-7802-46fc-9612-fd00544fa547","Type":"ContainerDied","Data":"b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1"}
Nov 01 04:13:55.639623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.639554    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq" event={"ID":"9975f6aa-7802-46fc-9612-fd00544fa547","Type":"ContainerDied","Data":"dd6c4601e9f4e6bc43aeaebfa1578c8621df35929a5118f1fb63fb7dd4e9db9b"}
Nov 01 04:13:55.639623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.639586    4882 scope.go:122] "RemoveContainer" containerID="b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1"
Nov 01 04:13:55.646347 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.646262    4882 generic.go:356] "Generic (PLEG): container finished" podID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerID="8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a" exitCode=0
Nov 01 04:13:55.647051 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.647024    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"
Nov 01 04:13:55.647179 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.647158    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" event={"ID":"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f","Type":"ContainerDied","Data":"8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a"}
Nov 01 04:13:55.647248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.647201    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb" event={"ID":"dc94dbdc-bb4f-4b4b-aa94-a516adfb041f","Type":"ContainerDied","Data":"e54b9a5a16a4763c1c6a560af0f098dd2dfc4826b98d9e663eaafb9237178e2b"}
Nov 01 04:13:55.661609 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.661503    4882 scope.go:122] "RemoveContainer" containerID="b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1"
Nov 01 04:13:55.661905 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:13:55.661884    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1\": container with ID starting with b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1 not found: ID does not exist" containerID="b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1"
Nov 01 04:13:55.661964 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.661917    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1"} err="failed to get container status \"b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1\": rpc error: code = NotFound desc = could not find container \"b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1\": container with ID starting with b8a53031fc33dbef4541c25a66b745717cc234a0f1bccfcdbca6c2e14bbe76f1 not found: ID does not exist"
Nov 01 04:13:55.662013 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.661936    4882 scope.go:122] "RemoveContainer" containerID="8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a"
Nov 01 04:13:55.672046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.672022    4882 scope.go:122] "RemoveContainer" containerID="91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d"
Nov 01 04:13:55.682299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.682277    4882 scope.go:122] "RemoveContainer" containerID="8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a"
Nov 01 04:13:55.682660 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:13:55.682637    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a\": container with ID starting with 8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a not found: ID does not exist" containerID="8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a"
Nov 01 04:13:55.682747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.682672    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a"} err="failed to get container status \"8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a\": rpc error: code = NotFound desc = could not find container \"8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a\": container with ID starting with 8572e9340ad61c4bf3e4dc1a823ca2decd6ccdf350509a913f4c361296c8de7a not found: ID does not exist"
Nov 01 04:13:55.682747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.682691    4882 scope.go:122] "RemoveContainer" containerID="91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d"
Nov 01 04:13:55.682989 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:13:55.682972    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d\": container with ID starting with 91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d not found: ID does not exist" containerID="91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d"
Nov 01 04:13:55.683043 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.682996    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d"} err="failed to get container status \"91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d\": rpc error: code = NotFound desc = could not find container \"91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d\": container with ID starting with 91c0599bc0fdc7909c0a4bf86ac42b86f145228c0a563b4a409550a1b7e7146d not found: ID does not exist"
Nov 01 04:13:55.867820 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.867708    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"]
Nov 01 04:13:55.872856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:55.872824    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vtwzq"]
Nov 01 04:13:56.091020 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:56.090981    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"]
Nov 01 04:13:56.095593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:56.095561    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bj4xb"]
Nov 01 04:13:57.604002 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:57.603957    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" path="/var/lib/kubelet/pods/9975f6aa-7802-46fc-9612-fd00544fa547/volumes"
Nov 01 04:13:57.604815 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:57.604792    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" path="/var/lib/kubelet/pods/dc94dbdc-bb4f-4b4b-aa94-a516adfb041f/volumes"
Nov 01 04:13:58.436326 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.436303    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 04:13:58.542958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.542919    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-envoy\") pod \"8e6df42f-9caf-4a86-a91a-2a51461a8645\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") "
Nov 01 04:13:58.543526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.543072    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6df42f-9caf-4a86-a91a-2a51461a8645-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6df42f-9caf-4a86-a91a-2a51461a8645-istiod-ca-cert\") pod \"8e6df42f-9caf-4a86-a91a-2a51461a8645\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") "
Nov 01 04:13:58.543526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.543111    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-podinfo\") pod \"8e6df42f-9caf-4a86-a91a-2a51461a8645\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") "
Nov 01 04:13:58.543526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.543148    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-token\" (UniqueName: \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-token\") pod \"8e6df42f-9caf-4a86-a91a-2a51461a8645\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") "
Nov 01 04:13:58.543526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.543227    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-kube-api-access-hrhwt\" (UniqueName: \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-kube-api-access-hrhwt\") pod \"8e6df42f-9caf-4a86-a91a-2a51461a8645\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") "
Nov 01 04:13:58.543526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.543300    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-data\") pod \"8e6df42f-9caf-4a86-a91a-2a51461a8645\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") "
Nov 01 04:13:58.543526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.543382    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-certs\") pod \"8e6df42f-9caf-4a86-a91a-2a51461a8645\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") "
Nov 01 04:13:58.543526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.543417    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-credential-socket\") pod \"8e6df42f-9caf-4a86-a91a-2a51461a8645\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") "
Nov 01 04:13:58.543526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.543468    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-socket\") pod \"8e6df42f-9caf-4a86-a91a-2a51461a8645\" (UID: \"8e6df42f-9caf-4a86-a91a-2a51461a8645\") "
Nov 01 04:13:58.544037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.544016    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-certs" pod "8e6df42f-9caf-4a86-a91a-2a51461a8645" (UID: "8e6df42f-9caf-4a86-a91a-2a51461a8645"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:58.544304 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.544267    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-socket" pod "8e6df42f-9caf-4a86-a91a-2a51461a8645" (UID: "8e6df42f-9caf-4a86-a91a-2a51461a8645"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:58.544473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.544367    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-credential-socket" pod "8e6df42f-9caf-4a86-a91a-2a51461a8645" (UID: "8e6df42f-9caf-4a86-a91a-2a51461a8645"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:58.544473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.544414    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-data" pod "8e6df42f-9caf-4a86-a91a-2a51461a8645" (UID: "8e6df42f-9caf-4a86-a91a-2a51461a8645"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:58.544617 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.544583    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6df42f-9caf-4a86-a91a-2a51461a8645-istiod-ca-cert" pod "8e6df42f-9caf-4a86-a91a-2a51461a8645" (UID: "8e6df42f-9caf-4a86-a91a-2a51461a8645"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:13:58.546703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.546662    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-envoy" pod "8e6df42f-9caf-4a86-a91a-2a51461a8645" (UID: "8e6df42f-9caf-4a86-a91a-2a51461a8645"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:13:58.546703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.546674    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-token" pod "8e6df42f-9caf-4a86-a91a-2a51461a8645" (UID: "8e6df42f-9caf-4a86-a91a-2a51461a8645"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:13:58.547048 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.547027    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-podinfo" pod "8e6df42f-9caf-4a86-a91a-2a51461a8645" (UID: "8e6df42f-9caf-4a86-a91a-2a51461a8645"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:13:58.547281 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.547262    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-kube-api-access-hrhwt" pod "8e6df42f-9caf-4a86-a91a-2a51461a8645" (UID: "8e6df42f-9caf-4a86-a91a-2a51461a8645"). InnerVolumeSpecName "kube-api-access-hrhwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:13:58.645524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.645488    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:58.645524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.645526    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:58.645966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.645542    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:58.645966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.645560    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:58.645966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.645577    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6df42f-9caf-4a86-a91a-2a51461a8645-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:58.645966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.645593    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:58.645966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.645607    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:58.645966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.645622    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrhwt\" (UniqueName: \"kubernetes.io/projected/8e6df42f-9caf-4a86-a91a-2a51461a8645-kube-api-access-hrhwt\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:58.645966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.645637    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6df42f-9caf-4a86-a91a-2a51461a8645-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:13:58.669367 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.669329    4882 generic.go:356] "Generic (PLEG): container finished" podID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerID="4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55" exitCode=0
Nov 01 04:13:58.669581 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.669387    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"
Nov 01 04:13:58.669581 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.669413    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" event={"ID":"8e6df42f-9caf-4a86-a91a-2a51461a8645","Type":"ContainerDied","Data":"4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55"}
Nov 01 04:13:58.669581 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.669514    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p" event={"ID":"8e6df42f-9caf-4a86-a91a-2a51461a8645","Type":"ContainerDied","Data":"ad765f569285779c2b055a5ff1ce48624812298615aa795dd49552d820f867b8"}
Nov 01 04:13:58.669581 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.669550    4882 scope.go:122] "RemoveContainer" containerID="4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55"
Nov 01 04:13:58.681723 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.681700    4882 scope.go:122] "RemoveContainer" containerID="4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55"
Nov 01 04:13:58.682059 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:13:58.682039    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55\": container with ID starting with 4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55 not found: ID does not exist" containerID="4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55"
Nov 01 04:13:58.682127 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.682068    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55"} err="failed to get container status \"4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55\": rpc error: code = NotFound desc = could not find container \"4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55\": container with ID starting with 4ea033e6f7175f0930fd5bdb6d7281ed5e0deafc2dc8ec85728f7f5d67e11e55 not found: ID does not exist"
Nov 01 04:13:58.727691 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:13:58.727606    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:13:58.824029 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.823985    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"]
Nov 01 04:13:58.828086 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:58.828058    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-j269p"]
Nov 01 04:13:59.600129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:13:59.600091    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" path="/var/lib/kubelet/pods/8e6df42f-9caf-4a86-a91a-2a51461a8645/volumes"
Nov 01 04:14:01.365057 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.365023    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"]
Nov 01 04:14:01.365606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.365588    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerName="istio-proxy"
Nov 01 04:14:01.365606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.365607    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerName="istio-proxy"
Nov 01 04:14:01.365747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.365616    4882 cpu_manager.go:414] "containerMap: removing container" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy"
Nov 01 04:14:01.365747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.365624    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy"
Nov 01 04:14:01.365747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.365677    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy"
Nov 01 04:14:01.365747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.365683    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy"
Nov 01 04:14:01.365932 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.365797    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy"
Nov 01 04:14:01.365932 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.365805    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6df42f-9caf-4a86-a91a-2a51461a8645" containerName="istio-proxy"
Nov 01 04:14:01.365932 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.365811    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy"
Nov 01 04:14:01.365932 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.365823    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9975f6aa-7802-46fc-9612-fd00544fa547" containerName="istio-proxy"
Nov 01 04:14:01.370837 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.370814    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:01.422553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.422407    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"]
Nov 01 04:14:01.464670 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.464626    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:14:01.464670 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.464671    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:14:01.493856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.493819    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlq6r\" (UniqueName: \"kubernetes.io/projected/773694d7-d894-490f-be9a-0a064f68d0db-kube-api-access-mlq6r\") pod \"maas-api-revoke-sub-e0a45508-5fx2d\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:01.494020 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.493999    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a45508-5fx2d\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:01.494069 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.494055    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a45508-5fx2d\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:01.595533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.595489    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a45508-5fx2d\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:01.595741 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.595576    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a45508-5fx2d\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:01.595816 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.595767    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlq6r\" (UniqueName: \"kubernetes.io/projected/773694d7-d894-490f-be9a-0a064f68d0db-kube-api-access-mlq6r\") pod \"maas-api-revoke-sub-e0a45508-5fx2d\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:01.596336 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.596312    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a45508-5fx2d\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:01.598070 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.598046    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a45508-5fx2d\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:01.653263 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.653188    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlq6r\" (UniqueName: \"kubernetes.io/projected/773694d7-d894-490f-be9a-0a064f68d0db-kube-api-access-mlq6r\") pod \"maas-api-revoke-sub-e0a45508-5fx2d\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:01.685949 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:01.685911    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:02.076460 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:02.076416    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"]
Nov 01 04:14:02.078975 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:14:02.078946    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773694d7_d894_490f_be9a_0a064f68d0db.slice/crio-7cf275b98dad3af1e2b16a2c7b703eaa9c6348b8f56b338ae22661e07eb11ada WatchSource:0}: Error finding container 7cf275b98dad3af1e2b16a2c7b703eaa9c6348b8f56b338ae22661e07eb11ada: Status 404 returned error can't find the container with id 7cf275b98dad3af1e2b16a2c7b703eaa9c6348b8f56b338ae22661e07eb11ada
Nov 01 04:14:02.701715 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:02.701677    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d" event={"ID":"773694d7-d894-490f-be9a-0a064f68d0db","Type":"ContainerStarted","Data":"dcddb239b1ef7cc6d1a1a8c77dcb6f89776cf00e0711be1ef033b7c0aae35353"}
Nov 01 04:14:02.702149 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:02.701735    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d" event={"ID":"773694d7-d894-490f-be9a-0a064f68d0db","Type":"ContainerStarted","Data":"7cf275b98dad3af1e2b16a2c7b703eaa9c6348b8f56b338ae22661e07eb11ada"}
Nov 01 04:14:02.811310 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:02.811253    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d" podStartSLOduration=1.811237362 podStartE2EDuration="1.811237362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:14:02.808526174 +0000 UTC m=+1911.370559191" watchObservedRunningTime="2026-11-01 04:14:02.811237362 +0000 UTC m=+1911.373270379"
Nov 01 04:14:03.712377 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:03.712333    4882 generic.go:356] "Generic (PLEG): container finished" podID="773694d7-d894-490f-be9a-0a064f68d0db" containerID="dcddb239b1ef7cc6d1a1a8c77dcb6f89776cf00e0711be1ef033b7c0aae35353" exitCode=0
Nov 01 04:14:03.712891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:03.712409    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d" event={"ID":"773694d7-d894-490f-be9a-0a064f68d0db","Type":"ContainerDied","Data":"dcddb239b1ef7cc6d1a1a8c77dcb6f89776cf00e0711be1ef033b7c0aae35353"}
Nov 01 04:14:04.869405 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:04.869373    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:04.942242 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:04.942206    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-xks-ca\") pod \"773694d7-d894-490f-be9a-0a064f68d0db\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") "
Nov 01 04:14:04.942406 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:04.942303    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-openshift-ca\") pod \"773694d7-d894-490f-be9a-0a064f68d0db\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") "
Nov 01 04:14:04.942406 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:04.942332    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773694d7-d894-490f-be9a-0a064f68d0db-kube-api-access-mlq6r\" (UniqueName: \"kubernetes.io/projected/773694d7-d894-490f-be9a-0a064f68d0db-kube-api-access-mlq6r\") pod \"773694d7-d894-490f-be9a-0a064f68d0db\" (UID: \"773694d7-d894-490f-be9a-0a064f68d0db\") "
Nov 01 04:14:04.942717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:04.942696    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-openshift-ca" pod "773694d7-d894-490f-be9a-0a064f68d0db" (UID: "773694d7-d894-490f-be9a-0a064f68d0db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:14:04.942954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:04.942935    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:14:04.944474 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:04.944449    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-xks-ca" pod "773694d7-d894-490f-be9a-0a064f68d0db" (UID: "773694d7-d894-490f-be9a-0a064f68d0db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:14:04.944592 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:04.944573    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773694d7-d894-490f-be9a-0a064f68d0db-kube-api-access-mlq6r" pod "773694d7-d894-490f-be9a-0a064f68d0db" (UID: "773694d7-d894-490f-be9a-0a064f68d0db"). InnerVolumeSpecName "kube-api-access-mlq6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:14:05.043984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:05.043898    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlq6r\" (UniqueName: \"kubernetes.io/projected/773694d7-d894-490f-be9a-0a064f68d0db-kube-api-access-mlq6r\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:14:05.043984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:05.043930    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/773694d7-d894-490f-be9a-0a064f68d0db-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:14:05.727101 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:05.727073    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"
Nov 01 04:14:05.727280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:05.727108    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d" event={"ID":"773694d7-d894-490f-be9a-0a064f68d0db","Type":"ContainerDied","Data":"7cf275b98dad3af1e2b16a2c7b703eaa9c6348b8f56b338ae22661e07eb11ada"}
Nov 01 04:14:05.727280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:05.727161    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cf275b98dad3af1e2b16a2c7b703eaa9c6348b8f56b338ae22661e07eb11ada"
Nov 01 04:14:13.038081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:13.038041    4882 scope.go:122] "RemoveContainer" containerID="5a551b3b5d48dc0478b99c75f36907b0e2289d1066f524391eab0d70e8b6e5d5"
Nov 01 04:14:13.055095 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:13.052672    4882 scope.go:122] "RemoveContainer" containerID="bfc40de66c245ff12fdb98e3901059a3796403c55f32850527296cc49e1ab9e5"
Nov 01 04:14:28.724315 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:14:28.724233    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:14:35.388390 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.388355    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"]
Nov 01 04:14:35.388913 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.388897    4882 cpu_manager.go:414] "containerMap: removing container" podUID="773694d7-d894-490f-be9a-0a064f68d0db" containerName="revoke-keys"
Nov 01 04:14:35.388962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.388913    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773694d7-d894-490f-be9a-0a064f68d0db" containerName="revoke-keys"
Nov 01 04:14:35.388962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.388957    4882 cpu_manager.go:414] "containerMap: removing container" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy"
Nov 01 04:14:35.389037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.388963    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc94dbdc-bb4f-4b4b-aa94-a516adfb041f" containerName="istio-proxy"
Nov 01 04:14:35.389091 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.389053    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="773694d7-d894-490f-be9a-0a064f68d0db" containerName="revoke-keys"
Nov 01 04:14:35.392620 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.392596    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:14:35.446781 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.446747    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5b71cd\""
Nov 01 04:14:35.447032 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.446841    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"]
Nov 01 04:14:35.447216 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.446947    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5b71cd-dockercfg-mz5qj\""
Nov 01 04:14:35.470964 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.470917    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbl46\" (UniqueName: \"kubernetes.io/projected/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-kube-api-access-qbl46\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z\" (UID: \"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:14:35.471254 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.471225    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z\" (UID: \"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:14:35.572553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.572513    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbl46\" (UniqueName: \"kubernetes.io/projected/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-kube-api-access-qbl46\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z\" (UID: \"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:14:35.572787 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.572707    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z\" (UID: \"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:14:35.573530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.573510    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z\" (UID: \"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:14:35.630031 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.630001    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbl46\" (UniqueName: \"kubernetes.io/projected/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-kube-api-access-qbl46\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z\" (UID: \"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:14:35.706510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:35.706476    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:14:36.085576 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.085553    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"]
Nov 01 04:14:36.087907 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:14:36.087877    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b64c7a6_19a3_4f3d_b771_4e3fb92af7fa.slice/crio-ab32fc65187fd609ff38380d7294ac555dfd5239b0c379214ec29a5d4b95796f WatchSource:0}: Error finding container ab32fc65187fd609ff38380d7294ac555dfd5239b0c379214ec29a5d4b95796f: Status 404 returned error can't find the container with id ab32fc65187fd609ff38380d7294ac555dfd5239b0c379214ec29a5d4b95796f
Nov 01 04:14:36.089876 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.089858    4882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 04:14:36.158584 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.158552    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"]
Nov 01 04:14:36.163775 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.163752    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:14:36.218776 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.218744    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"]
Nov 01 04:14:36.281185 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.281149    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpn97\" (UniqueName: \"kubernetes.io/projected/1c49bad6-a6a2-4766-8d4c-309d77e13eff-kube-api-access-cpn97\") pod \"payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr\" (UID: \"1c49bad6-a6a2-4766-8d4c-309d77e13eff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:14:36.281185 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.281194    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c49bad6-a6a2-4766-8d4c-309d77e13eff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr\" (UID: \"1c49bad6-a6a2-4766-8d4c-309d77e13eff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:14:36.382160 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.382066    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpn97\" (UniqueName: \"kubernetes.io/projected/1c49bad6-a6a2-4766-8d4c-309d77e13eff-kube-api-access-cpn97\") pod \"payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr\" (UID: \"1c49bad6-a6a2-4766-8d4c-309d77e13eff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:14:36.382160 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.382125    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c49bad6-a6a2-4766-8d4c-309d77e13eff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr\" (UID: \"1c49bad6-a6a2-4766-8d4c-309d77e13eff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:14:36.382943 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.382917    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c49bad6-a6a2-4766-8d4c-309d77e13eff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr\" (UID: \"1c49bad6-a6a2-4766-8d4c-309d77e13eff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:14:36.440040 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.440000    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpn97\" (UniqueName: \"kubernetes.io/projected/1c49bad6-a6a2-4766-8d4c-309d77e13eff-kube-api-access-cpn97\") pod \"payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr\" (UID: \"1c49bad6-a6a2-4766-8d4c-309d77e13eff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:14:36.477665 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.477628    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:14:36.861354 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:14:36.854051    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c49bad6_a6a2_4766_8d4c_309d77e13eff.slice/crio-46f5089147c6d35f07afb5d0f608382fc8a6911f5027e014869f0c1bb90b534c WatchSource:0}: Error finding container 46f5089147c6d35f07afb5d0f608382fc8a6911f5027e014869f0c1bb90b534c: Status 404 returned error can't find the container with id 46f5089147c6d35f07afb5d0f608382fc8a6911f5027e014869f0c1bb90b534c
Nov 01 04:14:36.861354 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:36.858595    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"]
Nov 01 04:14:37.000991 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:37.000940    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr" event={"ID":"1c49bad6-a6a2-4766-8d4c-309d77e13eff","Type":"ContainerStarted","Data":"46f5089147c6d35f07afb5d0f608382fc8a6911f5027e014869f0c1bb90b534c"}
Nov 01 04:14:37.002710 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:37.002676    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z" event={"ID":"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa","Type":"ContainerStarted","Data":"ab32fc65187fd609ff38380d7294ac555dfd5239b0c379214ec29a5d4b95796f"}
Nov 01 04:14:39.020677 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:39.020636    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z" event={"ID":"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa","Type":"ContainerStarted","Data":"3c8842398b7d6d0f7b634c3c14c06f2c27f9f6e907193c56003c9a31ac7aacaf"}
Nov 01 04:14:39.021167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:39.020730    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:14:39.022400 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:39.022372    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr" event={"ID":"1c49bad6-a6a2-4766-8d4c-309d77e13eff","Type":"ContainerStarted","Data":"40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04"}
Nov 01 04:14:39.022539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:39.022474    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:14:39.132086 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:39.132025    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z" podStartSLOduration=1.5656536399999998 podStartE2EDuration="4.132010056s" totalImagesPullingTime="2.566356416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:14:39.128701379 +0000 UTC m=+1947.690734397" watchObservedRunningTime="2026-11-01 04:14:39.132010056 +0000 UTC m=+1947.694043070"
Nov 01 04:14:39.238256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:39.238189    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr" podStartSLOduration=1.280217563 podStartE2EDuration="3.238169028s" totalImagesPullingTime="1.957951465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:14:39.236807032 +0000 UTC m=+1947.798840053" watchObservedRunningTime="2026-11-01 04:14:39.238169028 +0000 UTC m=+1947.800202045"
Nov 01 04:14:50.031414 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:50.031383    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:14:50.031916 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:14:50.031473    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:14:58.724596 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:14:58.724555    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:15:00.159035 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.159001    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"]
Nov 01 04:15:00.174462 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.172909    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"]
Nov 01 04:15:00.174462 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.173548    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"
Nov 01 04:15:00.188705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.188665    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"]
Nov 01 04:15:00.189010 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.188988    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"
Nov 01 04:15:00.195517 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.193446    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"
Nov 01 04:15:00.208470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.201687    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vg89\" (UniqueName: \"kubernetes.io/projected/bc5e01fe-46aa-4663-8788-c50f05c2d818-kube-api-access-2vg89\") pod \"maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp\" (UID: \"bc5e01fe-46aa-4663-8788-c50f05c2d818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"
Nov 01 04:15:00.208470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.201790    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6lp\" (UniqueName: \"kubernetes.io/projected/290b65f4-af57-4e95-b59e-414f4d2d9480-kube-api-access-qx6lp\") pod \"maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6\" (UID: \"290b65f4-af57-4e95-b59e-414f4d2d9480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"
Nov 01 04:15:00.208470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.201820    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d774ed2-7110-4cd5-b8e2-20de7406e20b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb\" (UID: \"4d774ed2-7110-4cd5-b8e2-20de7406e20b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"
Nov 01 04:15:00.208470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.201882    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rng\" (UniqueName: \"kubernetes.io/projected/4d774ed2-7110-4cd5-b8e2-20de7406e20b-kube-api-access-p6rng\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb\" (UID: \"4d774ed2-7110-4cd5-b8e2-20de7406e20b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"
Nov 01 04:15:00.208470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.201913    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc5e01fe-46aa-4663-8788-c50f05c2d818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp\" (UID: \"bc5e01fe-46aa-4663-8788-c50f05c2d818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"
Nov 01 04:15:00.208470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.201946    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/290b65f4-af57-4e95-b59e-414f4d2d9480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6\" (UID: \"290b65f4-af57-4e95-b59e-414f4d2d9480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"
Nov 01 04:15:00.208470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.205691    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"]
Nov 01 04:15:00.209874 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.209851    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"]
Nov 01 04:15:00.209994 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.209926    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"
Nov 01 04:15:00.216594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.216563    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"]
Nov 01 04:15:00.218321 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.218304    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"
Nov 01 04:15:00.222102 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.222082    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"]
Nov 01 04:15:00.222292 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.222179    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"
Nov 01 04:15:00.226095 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.226078    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"]
Nov 01 04:15:00.226247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.226228    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"
Nov 01 04:15:00.229973 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.229948    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"
Nov 01 04:15:00.247918 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.247892    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:15:00.260458 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.256456    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"]
Nov 01 04:15:00.260458 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.259800    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"]
Nov 01 04:15:00.308920 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.308878    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6lp\" (UniqueName: \"kubernetes.io/projected/290b65f4-af57-4e95-b59e-414f4d2d9480-kube-api-access-qx6lp\") pod \"maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6\" (UID: \"290b65f4-af57-4e95-b59e-414f4d2d9480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"
Nov 01 04:15:00.309118 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.308941    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d774ed2-7110-4cd5-b8e2-20de7406e20b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb\" (UID: \"4d774ed2-7110-4cd5-b8e2-20de7406e20b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"
Nov 01 04:15:00.309118 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.309004    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rng\" (UniqueName: \"kubernetes.io/projected/4d774ed2-7110-4cd5-b8e2-20de7406e20b-kube-api-access-p6rng\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb\" (UID: \"4d774ed2-7110-4cd5-b8e2-20de7406e20b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"
Nov 01 04:15:00.310677 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.310645    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc5e01fe-46aa-4663-8788-c50f05c2d818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp\" (UID: \"bc5e01fe-46aa-4663-8788-c50f05c2d818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"
Nov 01 04:15:00.310677 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.310714    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/290b65f4-af57-4e95-b59e-414f4d2d9480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6\" (UID: \"290b65f4-af57-4e95-b59e-414f4d2d9480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"
Nov 01 04:15:00.311332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.310930    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vg89\" (UniqueName: \"kubernetes.io/projected/bc5e01fe-46aa-4663-8788-c50f05c2d818-kube-api-access-2vg89\") pod \"maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp\" (UID: \"bc5e01fe-46aa-4663-8788-c50f05c2d818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"
Nov 01 04:15:00.311332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.311220    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d774ed2-7110-4cd5-b8e2-20de7406e20b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb\" (UID: \"4d774ed2-7110-4cd5-b8e2-20de7406e20b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"
Nov 01 04:15:00.315299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.312051    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc5e01fe-46aa-4663-8788-c50f05c2d818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp\" (UID: \"bc5e01fe-46aa-4663-8788-c50f05c2d818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"
Nov 01 04:15:00.315299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.312094    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/290b65f4-af57-4e95-b59e-414f4d2d9480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6\" (UID: \"290b65f4-af57-4e95-b59e-414f4d2d9480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"
Nov 01 04:15:00.315299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.312099    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"]
Nov 01 04:15:00.318340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.318316    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"]
Nov 01 04:15:00.322324 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.322310    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"]
Nov 01 04:15:00.327481 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.327317    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"]
Nov 01 04:15:00.394541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.394511    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rng\" (UniqueName: \"kubernetes.io/projected/4d774ed2-7110-4cd5-b8e2-20de7406e20b-kube-api-access-p6rng\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb\" (UID: \"4d774ed2-7110-4cd5-b8e2-20de7406e20b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"
Nov 01 04:15:00.400161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.399422    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vg89\" (UniqueName: \"kubernetes.io/projected/bc5e01fe-46aa-4663-8788-c50f05c2d818-kube-api-access-2vg89\") pod \"maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp\" (UID: \"bc5e01fe-46aa-4663-8788-c50f05c2d818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"
Nov 01 04:15:00.405528 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.405500    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"]
Nov 01 04:15:00.409675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.409607    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6lp\" (UniqueName: \"kubernetes.io/projected/290b65f4-af57-4e95-b59e-414f4d2d9480-kube-api-access-qx6lp\") pod \"maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6\" (UID: \"290b65f4-af57-4e95-b59e-414f4d2d9480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"
Nov 01 04:15:00.409805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.409677    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"]
Nov 01 04:15:00.414712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.414267    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad53e56-0eb2-49f3-8fea-7cea9e035732-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891775-dm9xs\" (UID: \"cad53e56-0eb2-49f3-8fea-7cea9e035732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"
Nov 01 04:15:00.414712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.414355    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69\" (UID: \"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"
Nov 01 04:15:00.414712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.414388    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsmpd\" (UniqueName: \"kubernetes.io/projected/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-kube-api-access-lsmpd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69\" (UID: \"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"
Nov 01 04:15:00.414712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.414480    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff2266b-1617-4268-b5e4-fe0d0c57b352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx\" (UID: \"6ff2266b-1617-4268-b5e4-fe0d0c57b352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"
Nov 01 04:15:00.414712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.414555    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da64a22b-d936-4d72-aac2-d3d9f8432671-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9\" (UID: \"da64a22b-d936-4d72-aac2-d3d9f8432671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"
Nov 01 04:15:00.415115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.415097    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnrt6\" (UniqueName: \"kubernetes.io/projected/6ff2266b-1617-4268-b5e4-fe0d0c57b352-kube-api-access-gnrt6\") pod \"maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx\" (UID: \"6ff2266b-1617-4268-b5e4-fe0d0c57b352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"
Nov 01 04:15:00.415202 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.415172    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59dlg\" (UniqueName: \"kubernetes.io/projected/da64a22b-d936-4d72-aac2-d3d9f8432671-kube-api-access-59dlg\") pod \"maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9\" (UID: \"da64a22b-d936-4d72-aac2-d3d9f8432671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"
Nov 01 04:15:00.415263 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.415219    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbkmr\" (UniqueName: \"kubernetes.io/projected/cad53e56-0eb2-49f3-8fea-7cea9e035732-kube-api-access-pbkmr\") pod \"maas-api-key-cleanup-29891775-dm9xs\" (UID: \"cad53e56-0eb2-49f3-8fea-7cea9e035732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"
Nov 01 04:15:00.415368 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.415349    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt\" (UID: \"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"
Nov 01 04:15:00.415471 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.415413    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqxxh\" (UniqueName: \"kubernetes.io/projected/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-kube-api-access-pqxxh\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt\" (UID: \"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"
Nov 01 04:15:00.503835 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.503801    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"
Nov 01 04:15:00.515529 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.515501    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"
Nov 01 04:15:00.516250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.516225    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqxxh\" (UniqueName: \"kubernetes.io/projected/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-kube-api-access-pqxxh\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt\" (UID: \"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"
Nov 01 04:15:00.516318 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.516272    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad53e56-0eb2-49f3-8fea-7cea9e035732-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891775-dm9xs\" (UID: \"cad53e56-0eb2-49f3-8fea-7cea9e035732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"
Nov 01 04:15:00.516355 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.516323    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69\" (UID: \"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"
Nov 01 04:15:00.516402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.516347    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsmpd\" (UniqueName: \"kubernetes.io/projected/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-kube-api-access-lsmpd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69\" (UID: \"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"
Nov 01 04:15:00.516465 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.516412    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff2266b-1617-4268-b5e4-fe0d0c57b352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx\" (UID: \"6ff2266b-1617-4268-b5e4-fe0d0c57b352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"
Nov 01 04:15:00.516531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.516494    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da64a22b-d936-4d72-aac2-d3d9f8432671-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9\" (UID: \"da64a22b-d936-4d72-aac2-d3d9f8432671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"
Nov 01 04:15:00.516591 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.516553    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnrt6\" (UniqueName: \"kubernetes.io/projected/6ff2266b-1617-4268-b5e4-fe0d0c57b352-kube-api-access-gnrt6\") pod \"maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx\" (UID: \"6ff2266b-1617-4268-b5e4-fe0d0c57b352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"
Nov 01 04:15:00.516646 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.516619    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59dlg\" (UniqueName: \"kubernetes.io/projected/da64a22b-d936-4d72-aac2-d3d9f8432671-kube-api-access-59dlg\") pod \"maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9\" (UID: \"da64a22b-d936-4d72-aac2-d3d9f8432671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"
Nov 01 04:15:00.516704 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.516658    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbkmr\" (UniqueName: \"kubernetes.io/projected/cad53e56-0eb2-49f3-8fea-7cea9e035732-kube-api-access-pbkmr\") pod \"maas-api-key-cleanup-29891775-dm9xs\" (UID: \"cad53e56-0eb2-49f3-8fea-7cea9e035732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"
Nov 01 04:15:00.516761 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.516713    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt\" (UID: \"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"
Nov 01 04:15:00.517489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.517459    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt\" (UID: \"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"
Nov 01 04:15:00.517636 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.517355    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff2266b-1617-4268-b5e4-fe0d0c57b352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx\" (UID: \"6ff2266b-1617-4268-b5e4-fe0d0c57b352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"
Nov 01 04:15:00.517731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.517706    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad53e56-0eb2-49f3-8fea-7cea9e035732-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891775-dm9xs\" (UID: \"cad53e56-0eb2-49f3-8fea-7cea9e035732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"
Nov 01 04:15:00.517805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.517720    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da64a22b-d936-4d72-aac2-d3d9f8432671-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9\" (UID: \"da64a22b-d936-4d72-aac2-d3d9f8432671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"
Nov 01 04:15:00.517866 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.517819    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69\" (UID: \"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"
Nov 01 04:15:00.527251 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.527220    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"
Nov 01 04:15:00.595863 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.595830    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqxxh\" (UniqueName: \"kubernetes.io/projected/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-kube-api-access-pqxxh\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt\" (UID: \"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"
Nov 01 04:15:00.597674 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.597621    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59dlg\" (UniqueName: \"kubernetes.io/projected/da64a22b-d936-4d72-aac2-d3d9f8432671-kube-api-access-59dlg\") pod \"maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9\" (UID: \"da64a22b-d936-4d72-aac2-d3d9f8432671\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"
Nov 01 04:15:00.599129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.599059    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbkmr\" (UniqueName: \"kubernetes.io/projected/cad53e56-0eb2-49f3-8fea-7cea9e035732-kube-api-access-pbkmr\") pod \"maas-api-key-cleanup-29891775-dm9xs\" (UID: \"cad53e56-0eb2-49f3-8fea-7cea9e035732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"
Nov 01 04:15:00.599794 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.599751    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnrt6\" (UniqueName: \"kubernetes.io/projected/6ff2266b-1617-4268-b5e4-fe0d0c57b352-kube-api-access-gnrt6\") pod \"maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx\" (UID: \"6ff2266b-1617-4268-b5e4-fe0d0c57b352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"
Nov 01 04:15:00.602136 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.602113    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsmpd\" (UniqueName: \"kubernetes.io/projected/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-kube-api-access-lsmpd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69\" (UID: \"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"
Nov 01 04:15:00.621672 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.620873    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"
Nov 01 04:15:00.849686 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.849647    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"
Nov 01 04:15:00.850501 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.850442    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"
Nov 01 04:15:00.875492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.871358    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"
Nov 01 04:15:00.894328 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.893101    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"
Nov 01 04:15:00.926622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:00.926415    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"]
Nov 01 04:15:01.039525 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:01.039418    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"]
Nov 01 04:15:01.053841 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:01.053808    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"]
Nov 01 04:15:01.083905 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:01.082728    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"]
Nov 01 04:15:01.232572 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:01.232477    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" event={"ID":"6ff2266b-1617-4268-b5e4-fe0d0c57b352","Type":"ContainerStarted","Data":"73b4713b55ae4d0d8820a23bd6b2cfb6e5ba00390393b7251f07648865ea5f4f"}
Nov 01 04:15:01.234073 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:01.234042    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" event={"ID":"4d774ed2-7110-4cd5-b8e2-20de7406e20b","Type":"ContainerStarted","Data":"b740eb958b77b0067e295ff67622389b544f4708b67d615d029a18e4d71fbf5b"}
Nov 01 04:15:01.235406 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:01.235382    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" event={"ID":"290b65f4-af57-4e95-b59e-414f4d2d9480","Type":"ContainerStarted","Data":"f1e5fe19030bb822cf40e69262394bff09a822a51633da651d3a4896f2c1c962"}
Nov 01 04:15:01.236637 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:01.236614    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp" event={"ID":"bc5e01fe-46aa-4663-8788-c50f05c2d818","Type":"ContainerStarted","Data":"a24008e5b84a08253940b5e9cf485d690454523d8f7261bf731a145f71fb78e7"}
Nov 01 04:15:01.262355 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:01.262329    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"]
Nov 01 04:15:01.266501 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:15:01.266476    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda64a22b_d936_4d72_aac2_d3d9f8432671.slice/crio-7074dcf15095ccd7e6365201820b29e2ad2f2e79f33fdc13bb2c78f5bf5e38a7 WatchSource:0}: Error finding container 7074dcf15095ccd7e6365201820b29e2ad2f2e79f33fdc13bb2c78f5bf5e38a7: Status 404 returned error can't find the container with id 7074dcf15095ccd7e6365201820b29e2ad2f2e79f33fdc13bb2c78f5bf5e38a7
Nov 01 04:15:01.294892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:01.294866    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"]
Nov 01 04:15:01.298576 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:15:01.298547    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e51db6_b4d2_4f75_99b9_1fd0a9c5dfef.slice/crio-1ee958d9aa9dedfbad768525eca62c68cac842af0888151d68be82ee7572d07e WatchSource:0}: Error finding container 1ee958d9aa9dedfbad768525eca62c68cac842af0888151d68be82ee7572d07e: Status 404 returned error can't find the container with id 1ee958d9aa9dedfbad768525eca62c68cac842af0888151d68be82ee7572d07e
Nov 01 04:15:01.353512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:01.353488    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"]
Nov 01 04:15:01.356251 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:15:01.356225    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad53e56_0eb2_49f3_8fea_7cea9e035732.slice/crio-e9decec73418dff06cd150bff93685b1ae405eb6a7504a423ef70150d91c08d7 WatchSource:0}: Error finding container e9decec73418dff06cd150bff93685b1ae405eb6a7504a423ef70150d91c08d7: Status 404 returned error can't find the container with id e9decec73418dff06cd150bff93685b1ae405eb6a7504a423ef70150d91c08d7
Nov 01 04:15:01.412750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:01.412664    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"]
Nov 01 04:15:01.467230 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:15:01.467203    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2fb0aa_2c54_489a_a22d_e2d25f014ee1.slice/crio-47a8e4c5ca724f6081b44a1f5f1c2fdf779fb449f18673ce930eabd1b96e1ff9 WatchSource:0}: Error finding container 47a8e4c5ca724f6081b44a1f5f1c2fdf779fb449f18673ce930eabd1b96e1ff9: Status 404 returned error can't find the container with id 47a8e4c5ca724f6081b44a1f5f1c2fdf779fb449f18673ce930eabd1b96e1ff9
Nov 01 04:15:02.247524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.247480    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" event={"ID":"6ff2266b-1617-4268-b5e4-fe0d0c57b352","Type":"ContainerStarted","Data":"45099031ce3520a2f5be997547460bcaa37a7d60d2e9465d11182d40381c3bbb"}
Nov 01 04:15:02.249486 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.249457    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" event={"ID":"da64a22b-d936-4d72-aac2-d3d9f8432671","Type":"ContainerStarted","Data":"4f18a16e3949a9025a981f5b175cecc1c662f1f6b8c7eed61101b628eabea0a5"}
Nov 01 04:15:02.249647 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.249511    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" event={"ID":"da64a22b-d936-4d72-aac2-d3d9f8432671","Type":"ContainerStarted","Data":"7074dcf15095ccd7e6365201820b29e2ad2f2e79f33fdc13bb2c78f5bf5e38a7"}
Nov 01 04:15:02.251505 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.251472    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" event={"ID":"4d774ed2-7110-4cd5-b8e2-20de7406e20b","Type":"ContainerStarted","Data":"6f01e3181684ea1c95b2f0c26ac0d00abe76dd125e74c78ff94f12b54d0390ac"}
Nov 01 04:15:02.253682 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.253654    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt" event={"ID":"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1","Type":"ContainerStarted","Data":"2e23e36f5ca51a48421ebaf1b084a3967aaa34eab18c967e79a383fa42e045c1"}
Nov 01 04:15:02.253779 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.253707    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt" event={"ID":"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1","Type":"ContainerStarted","Data":"47a8e4c5ca724f6081b44a1f5f1c2fdf779fb449f18673ce930eabd1b96e1ff9"}
Nov 01 04:15:02.256482 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.256459    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69" event={"ID":"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef","Type":"ContainerStarted","Data":"0a6c7ef2a1960f9986f3bbb6963262be99af8fc8ea19a371d99679f91b68dfbf"}
Nov 01 04:15:02.256591 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.256504    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69" event={"ID":"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef","Type":"ContainerStarted","Data":"1ee958d9aa9dedfbad768525eca62c68cac842af0888151d68be82ee7572d07e"}
Nov 01 04:15:02.258294 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.258272    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" event={"ID":"290b65f4-af57-4e95-b59e-414f4d2d9480","Type":"ContainerStarted","Data":"4e8d9bd45fa3357cc8b84a514c328abf7f8e74c33e88ee03610065a04ce6691b"}
Nov 01 04:15:02.260105 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.260081    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp" event={"ID":"bc5e01fe-46aa-4663-8788-c50f05c2d818","Type":"ContainerStarted","Data":"d36a7b96ed03bd5711468467648be6f3131de6c44b6a6823b5a5541952a6398a"}
Nov 01 04:15:02.268287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.267944    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" event={"ID":"cad53e56-0eb2-49f3-8fea-7cea9e035732","Type":"ContainerStarted","Data":"2a738cb6387866b59f5c9ad3171f72d8ee84039d1bf0c1d2b4a4b66a06a44b12"}
Nov 01 04:15:02.268287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.267995    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" event={"ID":"cad53e56-0eb2-49f3-8fea-7cea9e035732","Type":"ContainerStarted","Data":"e9decec73418dff06cd150bff93685b1ae405eb6a7504a423ef70150d91c08d7"}
Nov 01 04:15:02.359616 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.359506    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" podStartSLOduration=2.359491356 podStartE2EDuration="2.359491356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:02.355890228 +0000 UTC m=+1970.917923244" watchObservedRunningTime="2026-11-01 04:15:02.359491356 +0000 UTC m=+1970.921524370"
Nov 01 04:15:02.473638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.473584    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" podStartSLOduration=2.4735687 podStartE2EDuration="2.4735687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:02.47059632 +0000 UTC m=+1971.032629337" watchObservedRunningTime="2026-11-01 04:15:02.4735687 +0000 UTC m=+1971.035601742"
Nov 01 04:15:02.581786 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.581722    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt" podStartSLOduration=2.581706966 podStartE2EDuration="2.581706966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:02.57767803 +0000 UTC m=+1971.139711046" watchObservedRunningTime="2026-11-01 04:15:02.581706966 +0000 UTC m=+1971.143739980"
Nov 01 04:15:02.685254 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.685191    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" podStartSLOduration=2.685172272 podStartE2EDuration="2.685172272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:02.683974673 +0000 UTC m=+1971.246007692" watchObservedRunningTime="2026-11-01 04:15:02.685172272 +0000 UTC m=+1971.247205291"
Nov 01 04:15:02.792630 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.792565    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" podStartSLOduration=2.792549211 podStartE2EDuration="2.792549211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:02.791219492 +0000 UTC m=+1971.353252506" watchObservedRunningTime="2026-11-01 04:15:02.792549211 +0000 UTC m=+1971.354582227"
Nov 01 04:15:02.900199 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:02.900144    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" podStartSLOduration=2.900127051 podStartE2EDuration="2.900127051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:02.897988037 +0000 UTC m=+1971.460021053" watchObservedRunningTime="2026-11-01 04:15:02.900127051 +0000 UTC m=+1971.462160068"
Nov 01 04:15:02.943500 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:02.942690    4882 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2fb0aa_2c54_489a_a22d_e2d25f014ee1.slice/crio-conmon-2e23e36f5ca51a48421ebaf1b084a3967aaa34eab18c967e79a383fa42e045c1.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 04:15:03.006827 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.006770    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp" podStartSLOduration=3.006753668 podStartE2EDuration="3.006753668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:03.005192056 +0000 UTC m=+1971.567225074" watchObservedRunningTime="2026-11-01 04:15:03.006753668 +0000 UTC m=+1971.568786685"
Nov 01 04:15:03.114111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.114045    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69" podStartSLOduration=3.114027376 podStartE2EDuration="3.114027376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:03.111319665 +0000 UTC m=+1971.673352684" watchObservedRunningTime="2026-11-01 04:15:03.114027376 +0000 UTC m=+1971.676060392"
Nov 01 04:15:03.278454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.278344    4882 generic.go:356] "Generic (PLEG): container finished" podID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerID="2e23e36f5ca51a48421ebaf1b084a3967aaa34eab18c967e79a383fa42e045c1" exitCode=60
Nov 01 04:15:03.278454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.278439    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt" event={"ID":"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1","Type":"ContainerDied","Data":"2e23e36f5ca51a48421ebaf1b084a3967aaa34eab18c967e79a383fa42e045c1"}
Nov 01 04:15:03.279029 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.278852    4882 scope.go:122] "RemoveContainer" containerID="2e23e36f5ca51a48421ebaf1b084a3967aaa34eab18c967e79a383fa42e045c1"
Nov 01 04:15:03.281830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.281713    4882 generic.go:356] "Generic (PLEG): container finished" podID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerID="0a6c7ef2a1960f9986f3bbb6963262be99af8fc8ea19a371d99679f91b68dfbf" exitCode=60
Nov 01 04:15:03.281955 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.281933    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69" event={"ID":"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef","Type":"ContainerDied","Data":"0a6c7ef2a1960f9986f3bbb6963262be99af8fc8ea19a371d99679f91b68dfbf"}
Nov 01 04:15:03.282157 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.282106    4882 scope.go:122] "RemoveContainer" containerID="0a6c7ef2a1960f9986f3bbb6963262be99af8fc8ea19a371d99679f91b68dfbf"
Nov 01 04:15:03.284809 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.284751    4882 generic.go:356] "Generic (PLEG): container finished" podID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerID="4e8d9bd45fa3357cc8b84a514c328abf7f8e74c33e88ee03610065a04ce6691b" exitCode=60
Nov 01 04:15:03.284887 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.284828    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" event={"ID":"290b65f4-af57-4e95-b59e-414f4d2d9480","Type":"ContainerDied","Data":"4e8d9bd45fa3357cc8b84a514c328abf7f8e74c33e88ee03610065a04ce6691b"}
Nov 01 04:15:03.285260 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.285237    4882 scope.go:122] "RemoveContainer" containerID="4e8d9bd45fa3357cc8b84a514c328abf7f8e74c33e88ee03610065a04ce6691b"
Nov 01 04:15:03.287300 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.287274    4882 generic.go:356] "Generic (PLEG): container finished" podID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerID="d36a7b96ed03bd5711468467648be6f3131de6c44b6a6823b5a5541952a6398a" exitCode=60
Nov 01 04:15:03.287470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.287402    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp" event={"ID":"bc5e01fe-46aa-4663-8788-c50f05c2d818","Type":"ContainerDied","Data":"d36a7b96ed03bd5711468467648be6f3131de6c44b6a6823b5a5541952a6398a"}
Nov 01 04:15:03.287669 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.287653    4882 scope.go:122] "RemoveContainer" containerID="d36a7b96ed03bd5711468467648be6f3131de6c44b6a6823b5a5541952a6398a"
Nov 01 04:15:03.289931 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.289909    4882 generic.go:356] "Generic (PLEG): container finished" podID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerID="2a738cb6387866b59f5c9ad3171f72d8ee84039d1bf0c1d2b4a4b66a06a44b12" exitCode=60
Nov 01 04:15:03.290082 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.289985    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" event={"ID":"cad53e56-0eb2-49f3-8fea-7cea9e035732","Type":"ContainerDied","Data":"2a738cb6387866b59f5c9ad3171f72d8ee84039d1bf0c1d2b4a4b66a06a44b12"}
Nov 01 04:15:03.290259 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.290242    4882 scope.go:122] "RemoveContainer" containerID="2a738cb6387866b59f5c9ad3171f72d8ee84039d1bf0c1d2b4a4b66a06a44b12"
Nov 01 04:15:03.291875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.291850    4882 generic.go:356] "Generic (PLEG): container finished" podID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerID="45099031ce3520a2f5be997547460bcaa37a7d60d2e9465d11182d40381c3bbb" exitCode=60
Nov 01 04:15:03.292282 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.291979    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" event={"ID":"6ff2266b-1617-4268-b5e4-fe0d0c57b352","Type":"ContainerDied","Data":"45099031ce3520a2f5be997547460bcaa37a7d60d2e9465d11182d40381c3bbb"}
Nov 01 04:15:03.292282 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.292185    4882 scope.go:122] "RemoveContainer" containerID="45099031ce3520a2f5be997547460bcaa37a7d60d2e9465d11182d40381c3bbb"
Nov 01 04:15:03.295042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.295015    4882 generic.go:356] "Generic (PLEG): container finished" podID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerID="4f18a16e3949a9025a981f5b175cecc1c662f1f6b8c7eed61101b628eabea0a5" exitCode=60
Nov 01 04:15:03.295158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.295127    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" event={"ID":"da64a22b-d936-4d72-aac2-d3d9f8432671","Type":"ContainerDied","Data":"4f18a16e3949a9025a981f5b175cecc1c662f1f6b8c7eed61101b628eabea0a5"}
Nov 01 04:15:03.295794 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.295775    4882 scope.go:122] "RemoveContainer" containerID="4f18a16e3949a9025a981f5b175cecc1c662f1f6b8c7eed61101b628eabea0a5"
Nov 01 04:15:03.297255 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.297233    4882 generic.go:356] "Generic (PLEG): container finished" podID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerID="6f01e3181684ea1c95b2f0c26ac0d00abe76dd125e74c78ff94f12b54d0390ac" exitCode=60
Nov 01 04:15:03.297368 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.297292    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" event={"ID":"4d774ed2-7110-4cd5-b8e2-20de7406e20b","Type":"ContainerDied","Data":"6f01e3181684ea1c95b2f0c26ac0d00abe76dd125e74c78ff94f12b54d0390ac"}
Nov 01 04:15:03.297629 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:03.297612    4882 scope.go:122] "RemoveContainer" containerID="6f01e3181684ea1c95b2f0c26ac0d00abe76dd125e74c78ff94f12b54d0390ac"
Nov 01 04:15:04.306638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.306594    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" event={"ID":"6ff2266b-1617-4268-b5e4-fe0d0c57b352","Type":"ContainerStarted","Data":"d0edf66f644a407758daceda6d208a361a2c46a3aa2fa74c6a0486f58a118e9d"}
Nov 01 04:15:04.308784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.308754    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" event={"ID":"da64a22b-d936-4d72-aac2-d3d9f8432671","Type":"ContainerStarted","Data":"9e8b022c9ae6afea6d0d6a54ba3bbeaa1b6378add7e64e5e36ccee6a789968f5"}
Nov 01 04:15:04.311090 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.311063    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" event={"ID":"4d774ed2-7110-4cd5-b8e2-20de7406e20b","Type":"ContainerStarted","Data":"d1bb5df46040e001eb4bf8c418ce3d91ddb455eec161dc22b4b80ab5dcd6cebf"}
Nov 01 04:15:04.313292 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.313263    4882 generic.go:356] "Generic (PLEG): container finished" podID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerID="adf28bce4677bb11491a6466d28c8c50a254f61a52460bd747a8f5487e058459" exitCode=60
Nov 01 04:15:04.313444 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.313338    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt" event={"ID":"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1","Type":"ContainerDied","Data":"adf28bce4677bb11491a6466d28c8c50a254f61a52460bd747a8f5487e058459"}
Nov 01 04:15:04.313444 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.313412    4882 scope.go:122] "RemoveContainer" containerID="2e23e36f5ca51a48421ebaf1b084a3967aaa34eab18c967e79a383fa42e045c1"
Nov 01 04:15:04.313712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.313694    4882 scope.go:122] "RemoveContainer" containerID="adf28bce4677bb11491a6466d28c8c50a254f61a52460bd747a8f5487e058459"
Nov 01 04:15:04.314030 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:04.313997    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt_odh-ai-gateway-infra(0c2fb0aa-2c54-489a-a22d-e2d25f014ee1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1"
Nov 01 04:15:04.316303 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.316279    4882 generic.go:356] "Generic (PLEG): container finished" podID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerID="0e272c4490bd7e206ffc8713620814f12da4aaaf02d37be96e476cc69dfccacf" exitCode=60
Nov 01 04:15:04.316546 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.316524    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69" event={"ID":"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef","Type":"ContainerDied","Data":"0e272c4490bd7e206ffc8713620814f12da4aaaf02d37be96e476cc69dfccacf"}
Nov 01 04:15:04.316789 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.316772    4882 scope.go:122] "RemoveContainer" containerID="0e272c4490bd7e206ffc8713620814f12da4aaaf02d37be96e476cc69dfccacf"
Nov 01 04:15:04.317057 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:04.317036    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69_odh-ai-gateway-infra(38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef"
Nov 01 04:15:04.318845 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.318826    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" event={"ID":"290b65f4-af57-4e95-b59e-414f4d2d9480","Type":"ContainerStarted","Data":"991382b720eb78a518fbbc31cd29171c2ae0bc3cee4a50a6a2f0f91a1f029817"}
Nov 01 04:15:04.321749 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.321221    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp" event={"ID":"bc5e01fe-46aa-4663-8788-c50f05c2d818","Type":"ContainerStarted","Data":"8eeb9ec2c0ab28f575c21f730117487ab9e005599edb9677721fd201ad62e458"}
Nov 01 04:15:04.323772 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.323746    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" event={"ID":"cad53e56-0eb2-49f3-8fea-7cea9e035732","Type":"ContainerStarted","Data":"71e114fe89762fe3990449864315f408c36ebedb4eb5e88e9bac022264187a7e"}
Nov 01 04:15:04.333732 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:04.333703    4882 scope.go:122] "RemoveContainer" containerID="0a6c7ef2a1960f9986f3bbb6963262be99af8fc8ea19a371d99679f91b68dfbf"
Nov 01 04:15:05.338665 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.338556    4882 generic.go:356] "Generic (PLEG): container finished" podID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerID="498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc" exitCode=0
Nov 01 04:15:05.339462 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.338661    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" event={"ID":"2c4abcc8-b995-4851-9a00-5514e866ad89","Type":"ContainerDied","Data":"498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc"}
Nov 01 04:15:05.339462 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.339231    4882 scope.go:122] "RemoveContainer" containerID="498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc"
Nov 01 04:15:05.347846 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.347717    4882 scope.go:122] "RemoveContainer" containerID="adf28bce4677bb11491a6466d28c8c50a254f61a52460bd747a8f5487e058459"
Nov 01 04:15:05.348240 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.348203    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:15:05.348459 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.348422    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:15:05.348687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.348663    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:15:05.350166 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.349164    4882 scope.go:122] "RemoveContainer" containerID="0e272c4490bd7e206ffc8713620814f12da4aaaf02d37be96e476cc69dfccacf"
Nov 01 04:15:05.350166 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:05.349503    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69_odh-ai-gateway-infra(38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef"
Nov 01 04:15:05.350166 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:05.348488    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt_odh-ai-gateway-infra(0c2fb0aa-2c54-489a-a22d-e2d25f014ee1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1"
Nov 01 04:15:05.353187 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.353162    4882 generic.go:356] "Generic (PLEG): container finished" podID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerID="991382b720eb78a518fbbc31cd29171c2ae0bc3cee4a50a6a2f0f91a1f029817" exitCode=60
Nov 01 04:15:05.353475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.353422    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" event={"ID":"290b65f4-af57-4e95-b59e-414f4d2d9480","Type":"ContainerDied","Data":"991382b720eb78a518fbbc31cd29171c2ae0bc3cee4a50a6a2f0f91a1f029817"}
Nov 01 04:15:05.353551 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.353502    4882 scope.go:122] "RemoveContainer" containerID="4e8d9bd45fa3357cc8b84a514c328abf7f8e74c33e88ee03610065a04ce6691b"
Nov 01 04:15:05.353762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.353631    4882 scope.go:122] "RemoveContainer" containerID="991382b720eb78a518fbbc31cd29171c2ae0bc3cee4a50a6a2f0f91a1f029817"
Nov 01 04:15:05.353959 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:05.353935    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6_odh-ai-gateway-infra(290b65f4-af57-4e95-b59e-414f4d2d9480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480"
Nov 01 04:15:05.355962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.355941    4882 generic.go:356] "Generic (PLEG): container finished" podID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerID="8eeb9ec2c0ab28f575c21f730117487ab9e005599edb9677721fd201ad62e458" exitCode=60
Nov 01 04:15:05.356062 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.356018    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp" event={"ID":"bc5e01fe-46aa-4663-8788-c50f05c2d818","Type":"ContainerDied","Data":"8eeb9ec2c0ab28f575c21f730117487ab9e005599edb9677721fd201ad62e458"}
Nov 01 04:15:05.356266 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.356249    4882 scope.go:122] "RemoveContainer" containerID="8eeb9ec2c0ab28f575c21f730117487ab9e005599edb9677721fd201ad62e458"
Nov 01 04:15:05.356575 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:05.356552    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp_odh-ai-gateway-infra(bc5e01fe-46aa-4663-8788-c50f05c2d818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818"
Nov 01 04:15:05.358460 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.358419    4882 generic.go:356] "Generic (PLEG): container finished" podID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerID="71e114fe89762fe3990449864315f408c36ebedb4eb5e88e9bac022264187a7e" exitCode=60
Nov 01 04:15:05.358917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.358602    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" event={"ID":"cad53e56-0eb2-49f3-8fea-7cea9e035732","Type":"ContainerDied","Data":"71e114fe89762fe3990449864315f408c36ebedb4eb5e88e9bac022264187a7e"}
Nov 01 04:15:05.358917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.358770    4882 scope.go:122] "RemoveContainer" containerID="71e114fe89762fe3990449864315f408c36ebedb4eb5e88e9bac022264187a7e"
Nov 01 04:15:05.359082 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:05.359057    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891775-dm9xs_odh-ai-gateway-infra(cad53e56-0eb2-49f3-8fea-7cea9e035732)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732"
Nov 01 04:15:05.361422 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.361397    4882 generic.go:356] "Generic (PLEG): container finished" podID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerID="d0edf66f644a407758daceda6d208a361a2c46a3aa2fa74c6a0486f58a118e9d" exitCode=60
Nov 01 04:15:05.361703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.361679    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" event={"ID":"6ff2266b-1617-4268-b5e4-fe0d0c57b352","Type":"ContainerDied","Data":"d0edf66f644a407758daceda6d208a361a2c46a3aa2fa74c6a0486f58a118e9d"}
Nov 01 04:15:05.362184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.361825    4882 scope.go:122] "RemoveContainer" containerID="d0edf66f644a407758daceda6d208a361a2c46a3aa2fa74c6a0486f58a118e9d"
Nov 01 04:15:05.362500 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:05.362473    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx_odh-ai-gateway-infra(6ff2266b-1617-4268-b5e4-fe0d0c57b352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352"
Nov 01 04:15:05.365066 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.365042    4882 generic.go:356] "Generic (PLEG): container finished" podID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerID="9e8b022c9ae6afea6d0d6a54ba3bbeaa1b6378add7e64e5e36ccee6a789968f5" exitCode=60
Nov 01 04:15:05.365231 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.365146    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" event={"ID":"da64a22b-d936-4d72-aac2-d3d9f8432671","Type":"ContainerDied","Data":"9e8b022c9ae6afea6d0d6a54ba3bbeaa1b6378add7e64e5e36ccee6a789968f5"}
Nov 01 04:15:05.365368 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.365346    4882 scope.go:122] "RemoveContainer" containerID="9e8b022c9ae6afea6d0d6a54ba3bbeaa1b6378add7e64e5e36ccee6a789968f5"
Nov 01 04:15:05.365772 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:05.365746    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9_odh-ai-gateway-infra(da64a22b-d936-4d72-aac2-d3d9f8432671)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671"
Nov 01 04:15:05.367328 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.367305    4882 generic.go:356] "Generic (PLEG): container finished" podID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerID="d1bb5df46040e001eb4bf8c418ce3d91ddb455eec161dc22b4b80ab5dcd6cebf" exitCode=60
Nov 01 04:15:05.367463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.367379    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" event={"ID":"4d774ed2-7110-4cd5-b8e2-20de7406e20b","Type":"ContainerDied","Data":"d1bb5df46040e001eb4bf8c418ce3d91ddb455eec161dc22b4b80ab5dcd6cebf"}
Nov 01 04:15:05.367651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.367634    4882 scope.go:122] "RemoveContainer" containerID="d1bb5df46040e001eb4bf8c418ce3d91ddb455eec161dc22b4b80ab5dcd6cebf"
Nov 01 04:15:05.367912 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:05.367883    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb_odh-ai-gateway-infra(4d774ed2-7110-4cd5-b8e2-20de7406e20b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b"
Nov 01 04:15:05.390160 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.390141    4882 scope.go:122] "RemoveContainer" containerID="d36a7b96ed03bd5711468467648be6f3131de6c44b6a6823b5a5541952a6398a"
Nov 01 04:15:05.404984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.404957    4882 scope.go:122] "RemoveContainer" containerID="2a738cb6387866b59f5c9ad3171f72d8ee84039d1bf0c1d2b4a4b66a06a44b12"
Nov 01 04:15:05.418934 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.418911    4882 scope.go:122] "RemoveContainer" containerID="45099031ce3520a2f5be997547460bcaa37a7d60d2e9465d11182d40381c3bbb"
Nov 01 04:15:05.436076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.436051    4882 scope.go:122] "RemoveContainer" containerID="4f18a16e3949a9025a981f5b175cecc1c662f1f6b8c7eed61101b628eabea0a5"
Nov 01 04:15:05.450162 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:05.450140    4882 scope.go:122] "RemoveContainer" containerID="6f01e3181684ea1c95b2f0c26ac0d00abe76dd125e74c78ff94f12b54d0390ac"
Nov 01 04:15:06.398627 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:06.398588    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" event={"ID":"2c4abcc8-b995-4851-9a00-5514e866ad89","Type":"ContainerStarted","Data":"acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1"}
Nov 01 04:15:06.401028 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:06.401004    4882 scope.go:122] "RemoveContainer" containerID="991382b720eb78a518fbbc31cd29171c2ae0bc3cee4a50a6a2f0f91a1f029817"
Nov 01 04:15:06.401351 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:06.401324    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6_odh-ai-gateway-infra(290b65f4-af57-4e95-b59e-414f4d2d9480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480"
Nov 01 04:15:06.402765 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:06.402741    4882 scope.go:122] "RemoveContainer" containerID="8eeb9ec2c0ab28f575c21f730117487ab9e005599edb9677721fd201ad62e458"
Nov 01 04:15:06.403084 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:06.403057    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp_odh-ai-gateway-infra(bc5e01fe-46aa-4663-8788-c50f05c2d818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818"
Nov 01 04:15:06.404346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:06.404329    4882 scope.go:122] "RemoveContainer" containerID="71e114fe89762fe3990449864315f408c36ebedb4eb5e88e9bac022264187a7e"
Nov 01 04:15:06.404587 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:06.404569    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891775-dm9xs_odh-ai-gateway-infra(cad53e56-0eb2-49f3-8fea-7cea9e035732)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732"
Nov 01 04:15:06.406339 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:06.406319    4882 scope.go:122] "RemoveContainer" containerID="d0edf66f644a407758daceda6d208a361a2c46a3aa2fa74c6a0486f58a118e9d"
Nov 01 04:15:06.406630 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:06.406608    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx_odh-ai-gateway-infra(6ff2266b-1617-4268-b5e4-fe0d0c57b352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352"
Nov 01 04:15:06.407881 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:06.407859    4882 scope.go:122] "RemoveContainer" containerID="9e8b022c9ae6afea6d0d6a54ba3bbeaa1b6378add7e64e5e36ccee6a789968f5"
Nov 01 04:15:06.408132 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:06.408110    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9_odh-ai-gateway-infra(da64a22b-d936-4d72-aac2-d3d9f8432671)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671"
Nov 01 04:15:06.409309 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:06.409293    4882 scope.go:122] "RemoveContainer" containerID="d1bb5df46040e001eb4bf8c418ce3d91ddb455eec161dc22b4b80ab5dcd6cebf"
Nov 01 04:15:06.409501 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:06.409485    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb_odh-ai-gateway-infra(4d774ed2-7110-4cd5-b8e2-20de7406e20b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b"
Nov 01 04:15:06.526502 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:06.526470    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:15:06.527720 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:06.527688    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 04:15:06.527859 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:06.527733    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 04:15:07.452603 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:07.452562    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:15:07.526600 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:07.526562    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 04:15:07.526786 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:07.526617    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 04:15:08.526268 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:08.526233    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 04:15:08.526748 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:08.526291    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 04:15:09.329737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.329699    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"]
Nov 01 04:15:09.336476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.336454    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:09.385450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.385352    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:15:09.435382 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.435332    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"]
Nov 01 04:15:09.456549 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.456511    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cab5f60-n64lt\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:09.456757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.456559    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzczk\" (UniqueName: \"kubernetes.io/projected/e38030cd-cbfc-4729-b270-e456184ec6d4-kube-api-access-pzczk\") pod \"maas-api-revoke-sub-1cab5f60-n64lt\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:09.456757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.456668    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cab5f60-n64lt\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:09.527043 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.527004    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 04:15:09.527536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.527065    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 04:15:09.558354 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.558307    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cab5f60-n64lt\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:09.558354 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.558354    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzczk\" (UniqueName: \"kubernetes.io/projected/e38030cd-cbfc-4729-b270-e456184ec6d4-kube-api-access-pzczk\") pod \"maas-api-revoke-sub-1cab5f60-n64lt\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:09.558662 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.558394    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cab5f60-n64lt\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:09.559085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.559065    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cab5f60-n64lt\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:09.560969 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.560941    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cab5f60-n64lt\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:09.615933 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.615851    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzczk\" (UniqueName: \"kubernetes.io/projected/e38030cd-cbfc-4729-b270-e456184ec6d4-kube-api-access-pzczk\") pod \"maas-api-revoke-sub-1cab5f60-n64lt\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:09.650764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:09.650726    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:10.049950 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:10.049914    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"]
Nov 01 04:15:10.051139 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:15:10.051111    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38030cd_cbfc_4729_b270_e456184ec6d4.slice/crio-71dc50f943dc12e20b0c19bdc2c451fcf115dba109a9a861578440719509a2aa WatchSource:0}: Error finding container 71dc50f943dc12e20b0c19bdc2c451fcf115dba109a9a861578440719509a2aa: Status 404 returned error can't find the container with id 71dc50f943dc12e20b0c19bdc2c451fcf115dba109a9a861578440719509a2aa
Nov 01 04:15:10.439737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:10.439694    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt" event={"ID":"e38030cd-cbfc-4729-b270-e456184ec6d4","Type":"ContainerStarted","Data":"71dc50f943dc12e20b0c19bdc2c451fcf115dba109a9a861578440719509a2aa"}
Nov 01 04:15:10.526600 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:10.526572    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 04:15:10.526731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:10.526620    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 04:15:11.448004 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:11.447969    4882 generic.go:356] "Generic (PLEG): container finished" podID="e38030cd-cbfc-4729-b270-e456184ec6d4" containerID="97e4e08a0e7eb3633f4b50ef0c24509001eb163c1eb9448ef0b2be1314646662" exitCode=0
Nov 01 04:15:11.448398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:11.448065    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt" event={"ID":"e38030cd-cbfc-4729-b270-e456184ec6d4","Type":"ContainerDied","Data":"97e4e08a0e7eb3633f4b50ef0c24509001eb163c1eb9448ef0b2be1314646662"}
Nov 01 04:15:11.526851 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:11.526819    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 04:15:11.527030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:11.526872    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 04:15:12.533896 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.533552    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:15:12.535901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.535838    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:15:12.623269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.623244    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:12.721003 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.720913    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-openshift-ca\") pod \"e38030cd-cbfc-4729-b270-e456184ec6d4\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") "
Nov 01 04:15:12.721189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.721134    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-xks-ca\") pod \"e38030cd-cbfc-4729-b270-e456184ec6d4\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") "
Nov 01 04:15:12.721269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.721205    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38030cd-cbfc-4729-b270-e456184ec6d4-kube-api-access-pzczk\" (UniqueName: \"kubernetes.io/projected/e38030cd-cbfc-4729-b270-e456184ec6d4-kube-api-access-pzczk\") pod \"e38030cd-cbfc-4729-b270-e456184ec6d4\" (UID: \"e38030cd-cbfc-4729-b270-e456184ec6d4\") "
Nov 01 04:15:12.721477 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.721341    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-openshift-ca" pod "e38030cd-cbfc-4729-b270-e456184ec6d4" (UID: "e38030cd-cbfc-4729-b270-e456184ec6d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:12.721740 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.721722    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:12.723803 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.723760    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-xks-ca" pod "e38030cd-cbfc-4729-b270-e456184ec6d4" (UID: "e38030cd-cbfc-4729-b270-e456184ec6d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:15:12.723909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.723824    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38030cd-cbfc-4729-b270-e456184ec6d4-kube-api-access-pzczk" pod "e38030cd-cbfc-4729-b270-e456184ec6d4" (UID: "e38030cd-cbfc-4729-b270-e456184ec6d4"). InnerVolumeSpecName "kube-api-access-pzczk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:12.827471 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.823172    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38030cd-cbfc-4729-b270-e456184ec6d4-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:12.827471 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:12.823203    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzczk\" (UniqueName: \"kubernetes.io/projected/e38030cd-cbfc-4729-b270-e456184ec6d4-kube-api-access-pzczk\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:13.101937 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:13.101730    4882 scope.go:122] "RemoveContainer" containerID="1ef46474d2af463e5ed69269b63511f655fbefa9dba7ba3755b7add36699b6df"
Nov 01 04:15:13.463153 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:13.463119    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"
Nov 01 04:15:13.463326 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:13.463149    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt" event={"ID":"e38030cd-cbfc-4729-b270-e456184ec6d4","Type":"ContainerDied","Data":"71dc50f943dc12e20b0c19bdc2c451fcf115dba109a9a861578440719509a2aa"}
Nov 01 04:15:13.463326 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:13.463199    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71dc50f943dc12e20b0c19bdc2c451fcf115dba109a9a861578440719509a2aa"
Nov 01 04:15:14.592885 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:14.592855    4882 scope.go:122] "RemoveContainer" containerID="0e272c4490bd7e206ffc8713620814f12da4aaaf02d37be96e476cc69dfccacf"
Nov 01 04:15:15.479010 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:15.478969    4882 generic.go:356] "Generic (PLEG): container finished" podID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerID="08c11b65c41aba97b29711e86af9c2a8a47c9b3dcee05d72b7da25affe526b7c" exitCode=60
Nov 01 04:15:15.479202 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:15.479039    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69" event={"ID":"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef","Type":"ContainerDied","Data":"08c11b65c41aba97b29711e86af9c2a8a47c9b3dcee05d72b7da25affe526b7c"}
Nov 01 04:15:15.479202 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:15.479101    4882 scope.go:122] "RemoveContainer" containerID="0e272c4490bd7e206ffc8713620814f12da4aaaf02d37be96e476cc69dfccacf"
Nov 01 04:15:15.479553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:15.479536    4882 scope.go:122] "RemoveContainer" containerID="08c11b65c41aba97b29711e86af9c2a8a47c9b3dcee05d72b7da25affe526b7c"
Nov 01 04:15:15.479922 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:15.479887    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69_odh-ai-gateway-infra(38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef"
Nov 01 04:15:15.593002 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:15.592960    4882 scope.go:122] "RemoveContainer" containerID="9e8b022c9ae6afea6d0d6a54ba3bbeaa1b6378add7e64e5e36ccee6a789968f5"
Nov 01 04:15:16.530855 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:16.529509    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" event={"ID":"da64a22b-d936-4d72-aac2-d3d9f8432671","Type":"ContainerStarted","Data":"1e02cab72c5872823726dd9ca539d415ba55147173edb4f0f510be4dad663014"}
Nov 01 04:15:16.614541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:16.594798    4882 scope.go:122] "RemoveContainer" containerID="991382b720eb78a518fbbc31cd29171c2ae0bc3cee4a50a6a2f0f91a1f029817"
Nov 01 04:15:16.663514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:16.663480    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"]
Nov 01 04:15:16.853582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:16.853552    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"
Nov 01 04:15:16.900952 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:16.900920    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-kube-api-access-lsmpd\" (UniqueName: \"kubernetes.io/projected/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-kube-api-access-lsmpd\") pod \"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef\" (UID: \"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef\") "
Nov 01 04:15:16.901114 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:16.901078    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-maas-api-service-ca\") pod \"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef\" (UID: \"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef\") "
Nov 01 04:15:16.901445 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:16.901401    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-maas-api-service-ca" pod "38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" (UID: "38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:16.903089 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:16.903066    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-kube-api-access-lsmpd" pod "38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" (UID: "38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef"). InnerVolumeSpecName "kube-api-access-lsmpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:17.002286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.002249    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:17.002286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.002280    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsmpd\" (UniqueName: \"kubernetes.io/projected/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef-kube-api-access-lsmpd\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:17.541817 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.541783    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"
Nov 01 04:15:17.541817 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.541802    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69" event={"ID":"38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef","Type":"ContainerDied","Data":"1ee958d9aa9dedfbad768525eca62c68cac842af0888151d68be82ee7572d07e"}
Nov 01 04:15:17.542089 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.541889    4882 scope.go:122] "RemoveContainer" containerID="08c11b65c41aba97b29711e86af9c2a8a47c9b3dcee05d72b7da25affe526b7c"
Nov 01 04:15:17.547350 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.547325    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" event={"ID":"290b65f4-af57-4e95-b59e-414f4d2d9480","Type":"ContainerStarted","Data":"772bc78184d55c1a3de7e2fb82df651b8590283dac35348fed88e9f53894b198"}
Nov 01 04:15:17.560308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.560279    4882 generic.go:356] "Generic (PLEG): container finished" podID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerID="1e02cab72c5872823726dd9ca539d415ba55147173edb4f0f510be4dad663014" exitCode=60
Nov 01 04:15:17.560418 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.560343    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" event={"ID":"da64a22b-d936-4d72-aac2-d3d9f8432671","Type":"ContainerDied","Data":"1e02cab72c5872823726dd9ca539d415ba55147173edb4f0f510be4dad663014"}
Nov 01 04:15:17.560491 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.560417    4882 scope.go:122] "RemoveContainer" containerID="9e8b022c9ae6afea6d0d6a54ba3bbeaa1b6378add7e64e5e36ccee6a789968f5"
Nov 01 04:15:17.560804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.560787    4882 scope.go:122] "RemoveContainer" containerID="1e02cab72c5872823726dd9ca539d415ba55147173edb4f0f510be4dad663014"
Nov 01 04:15:17.561180 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:17.561153    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9_odh-ai-gateway-infra(da64a22b-d936-4d72-aac2-d3d9f8432671)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671"
Nov 01 04:15:17.593234 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.593208    4882 scope.go:122] "RemoveContainer" containerID="8eeb9ec2c0ab28f575c21f730117487ab9e005599edb9677721fd201ad62e458"
Nov 01 04:15:17.655077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.655051    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"]
Nov 01 04:15:17.695489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.695463    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"]
Nov 01 04:15:17.699616 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:17.699589    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891775-84d69"]
Nov 01 04:15:18.577685 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.577651    4882 generic.go:356] "Generic (PLEG): container finished" podID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerID="772bc78184d55c1a3de7e2fb82df651b8590283dac35348fed88e9f53894b198" exitCode=60
Nov 01 04:15:18.577848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.577728    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" event={"ID":"290b65f4-af57-4e95-b59e-414f4d2d9480","Type":"ContainerDied","Data":"772bc78184d55c1a3de7e2fb82df651b8590283dac35348fed88e9f53894b198"}
Nov 01 04:15:18.577848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.577806    4882 scope.go:122] "RemoveContainer" containerID="991382b720eb78a518fbbc31cd29171c2ae0bc3cee4a50a6a2f0f91a1f029817"
Nov 01 04:15:18.578316 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.578269    4882 scope.go:122] "RemoveContainer" containerID="772bc78184d55c1a3de7e2fb82df651b8590283dac35348fed88e9f53894b198"
Nov 01 04:15:18.578808 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:18.578784    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6_odh-ai-gateway-infra(290b65f4-af57-4e95-b59e-414f4d2d9480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480"
Nov 01 04:15:18.580575 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.580557    4882 generic.go:356] "Generic (PLEG): container finished" podID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerID="ad81f9411a19e9ea6557546357dc1ce582467c214a2ae3bde2a2c1a752775450" exitCode=60
Nov 01 04:15:18.580680 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.580630    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp" event={"ID":"bc5e01fe-46aa-4663-8788-c50f05c2d818","Type":"ContainerDied","Data":"ad81f9411a19e9ea6557546357dc1ce582467c214a2ae3bde2a2c1a752775450"}
Nov 01 04:15:18.581078 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.581060    4882 scope.go:122] "RemoveContainer" containerID="ad81f9411a19e9ea6557546357dc1ce582467c214a2ae3bde2a2c1a752775450"
Nov 01 04:15:18.581404 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:18.581384    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp_odh-ai-gateway-infra(bc5e01fe-46aa-4663-8788-c50f05c2d818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818"
Nov 01 04:15:18.592945 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.592921    4882 scope.go:122] "RemoveContainer" containerID="8eeb9ec2c0ab28f575c21f730117487ab9e005599edb9677721fd201ad62e458"
Nov 01 04:15:18.593064 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.593028    4882 scope.go:122] "RemoveContainer" containerID="71e114fe89762fe3990449864315f408c36ebedb4eb5e88e9bac022264187a7e"
Nov 01 04:15:18.593273 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.593254    4882 scope.go:122] "RemoveContainer" containerID="d1bb5df46040e001eb4bf8c418ce3d91ddb455eec161dc22b4b80ab5dcd6cebf"
Nov 01 04:15:18.593341 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.593315    4882 scope.go:122] "RemoveContainer" containerID="adf28bce4677bb11491a6466d28c8c50a254f61a52460bd747a8f5487e058459"
Nov 01 04:15:18.593964 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.593945    4882 scope.go:122] "RemoveContainer" containerID="d0edf66f644a407758daceda6d208a361a2c46a3aa2fa74c6a0486f58a118e9d"
Nov 01 04:15:18.744523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.744455    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"
Nov 01 04:15:18.814371 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.814338    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"]
Nov 01 04:15:18.825627 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.825600    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da64a22b-d936-4d72-aac2-d3d9f8432671-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da64a22b-d936-4d72-aac2-d3d9f8432671-maas-api-service-ca\") pod \"da64a22b-d936-4d72-aac2-d3d9f8432671\" (UID: \"da64a22b-d936-4d72-aac2-d3d9f8432671\") "
Nov 01 04:15:18.825763 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.825656    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da64a22b-d936-4d72-aac2-d3d9f8432671-kube-api-access-59dlg\" (UniqueName: \"kubernetes.io/projected/da64a22b-d936-4d72-aac2-d3d9f8432671-kube-api-access-59dlg\") pod \"da64a22b-d936-4d72-aac2-d3d9f8432671\" (UID: \"da64a22b-d936-4d72-aac2-d3d9f8432671\") "
Nov 01 04:15:18.826018 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.825991    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da64a22b-d936-4d72-aac2-d3d9f8432671-maas-api-service-ca" pod "da64a22b-d936-4d72-aac2-d3d9f8432671" (UID: "da64a22b-d936-4d72-aac2-d3d9f8432671"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:18.826310 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.826294    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da64a22b-d936-4d72-aac2-d3d9f8432671-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:18.863268 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.863238    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da64a22b-d936-4d72-aac2-d3d9f8432671-kube-api-access-59dlg" pod "da64a22b-d936-4d72-aac2-d3d9f8432671" (UID: "da64a22b-d936-4d72-aac2-d3d9f8432671"). InnerVolumeSpecName "kube-api-access-59dlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:18.928114 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:18.928083    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59dlg\" (UniqueName: \"kubernetes.io/projected/da64a22b-d936-4d72-aac2-d3d9f8432671-kube-api-access-59dlg\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:19.607366 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:19.607327    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"
Nov 01 04:15:19.608513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:19.608480    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" path="/var/lib/kubelet/pods/38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef/volumes"
Nov 01 04:15:19.609162 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:19.609134    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" event={"ID":"cad53e56-0eb2-49f3-8fea-7cea9e035732","Type":"ContainerStarted","Data":"de2a2a72481f12d76a0687d8320f4c594c94e9c78a01c0bd87426f2f9553510d"}
Nov 01 04:15:19.609301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:19.609187    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" event={"ID":"6ff2266b-1617-4268-b5e4-fe0d0c57b352","Type":"ContainerStarted","Data":"534b2b60789ba981d371645f607182e8ed76e889906a02606c1611e4081d2365"}
Nov 01 04:15:19.609301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:19.609219    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9" event={"ID":"da64a22b-d936-4d72-aac2-d3d9f8432671","Type":"ContainerDied","Data":"7074dcf15095ccd7e6365201820b29e2ad2f2e79f33fdc13bb2c78f5bf5e38a7"}
Nov 01 04:15:19.609301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:19.609246    4882 scope.go:122] "RemoveContainer" containerID="1e02cab72c5872823726dd9ca539d415ba55147173edb4f0f510be4dad663014"
Nov 01 04:15:19.610473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:19.610383    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" event={"ID":"4d774ed2-7110-4cd5-b8e2-20de7406e20b","Type":"ContainerStarted","Data":"6c0c37e14b744883973856367d78df36deed4e7abca8189421f9f04d8d5bfc0d"}
Nov 01 04:15:19.613549 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:19.613523    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt" event={"ID":"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1","Type":"ContainerStarted","Data":"9e8675ccb1668fdbd02b3782db6aab317c870ccd860c3f72dbf2b891f01ea138"}
Nov 01 04:15:19.803856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:19.803823    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"]
Nov 01 04:15:19.941905 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:19.941875    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"
Nov 01 04:15:19.950413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:19.950388    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"
Nov 01 04:15:20.044584 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.044550    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290b65f4-af57-4e95-b59e-414f4d2d9480-kube-api-access-qx6lp\" (UniqueName: \"kubernetes.io/projected/290b65f4-af57-4e95-b59e-414f4d2d9480-kube-api-access-qx6lp\") pod \"290b65f4-af57-4e95-b59e-414f4d2d9480\" (UID: \"290b65f4-af57-4e95-b59e-414f4d2d9480\") "
Nov 01 04:15:20.044760 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.044720    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc5e01fe-46aa-4663-8788-c50f05c2d818-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc5e01fe-46aa-4663-8788-c50f05c2d818-maas-api-service-ca\") pod \"bc5e01fe-46aa-4663-8788-c50f05c2d818\" (UID: \"bc5e01fe-46aa-4663-8788-c50f05c2d818\") "
Nov 01 04:15:20.044806 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.044758    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290b65f4-af57-4e95-b59e-414f4d2d9480-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/290b65f4-af57-4e95-b59e-414f4d2d9480-maas-api-service-ca\") pod \"290b65f4-af57-4e95-b59e-414f4d2d9480\" (UID: \"290b65f4-af57-4e95-b59e-414f4d2d9480\") "
Nov 01 04:15:20.044926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.044910    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc5e01fe-46aa-4663-8788-c50f05c2d818-kube-api-access-2vg89\" (UniqueName: \"kubernetes.io/projected/bc5e01fe-46aa-4663-8788-c50f05c2d818-kube-api-access-2vg89\") pod \"bc5e01fe-46aa-4663-8788-c50f05c2d818\" (UID: \"bc5e01fe-46aa-4663-8788-c50f05c2d818\") "
Nov 01 04:15:20.045115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.045092    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290b65f4-af57-4e95-b59e-414f4d2d9480-maas-api-service-ca" pod "290b65f4-af57-4e95-b59e-414f4d2d9480" (UID: "290b65f4-af57-4e95-b59e-414f4d2d9480"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:20.045184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.045092    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc5e01fe-46aa-4663-8788-c50f05c2d818-maas-api-service-ca" pod "bc5e01fe-46aa-4663-8788-c50f05c2d818" (UID: "bc5e01fe-46aa-4663-8788-c50f05c2d818"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:20.045414 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.045394    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc5e01fe-46aa-4663-8788-c50f05c2d818-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:20.045502 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.045415    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/290b65f4-af57-4e95-b59e-414f4d2d9480-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:20.046880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.046859    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290b65f4-af57-4e95-b59e-414f4d2d9480-kube-api-access-qx6lp" pod "290b65f4-af57-4e95-b59e-414f4d2d9480" (UID: "290b65f4-af57-4e95-b59e-414f4d2d9480"). InnerVolumeSpecName "kube-api-access-qx6lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:20.046959 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.046932    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5e01fe-46aa-4663-8788-c50f05c2d818-kube-api-access-2vg89" pod "bc5e01fe-46aa-4663-8788-c50f05c2d818" (UID: "bc5e01fe-46aa-4663-8788-c50f05c2d818"). InnerVolumeSpecName "kube-api-access-2vg89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:20.147375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.147286    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vg89\" (UniqueName: \"kubernetes.io/projected/bc5e01fe-46aa-4663-8788-c50f05c2d818-kube-api-access-2vg89\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:20.147375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.147316    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx6lp\" (UniqueName: \"kubernetes.io/projected/290b65f4-af57-4e95-b59e-414f4d2d9480-kube-api-access-qx6lp\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:20.157995 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.157960    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"]
Nov 01 04:15:20.158918 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.158894    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e38030cd-cbfc-4729-b270-e456184ec6d4" containerName="revoke-keys"
Nov 01 04:15:20.158918 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.158920    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38030cd-cbfc-4729-b270-e456184ec6d4" containerName="revoke-keys"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.158941    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerName="cleanup"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.158949    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerName="cleanup"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.158964    4882 cpu_manager.go:414] "containerMap: removing container" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerName="cleanup"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.158971    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerName="cleanup"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.158986    4882 cpu_manager.go:414] "containerMap: removing container" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerName="cleanup"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.158993    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerName="cleanup"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159005    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerName="cleanup"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159012    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerName="cleanup"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159025    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerName="cleanup"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159032    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerName="cleanup"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159048    4882 cpu_manager.go:414] "containerMap: removing container" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerName="cleanup"
Nov 01 04:15:20.159056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159054    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159067    4882 cpu_manager.go:414] "containerMap: removing container" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159074    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159090    4882 cpu_manager.go:414] "containerMap: removing container" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159096    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159110    4882 cpu_manager.go:414] "containerMap: removing container" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159117    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159137    4882 cpu_manager.go:414] "containerMap: removing container" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159143    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159327    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159348    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159366    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159382    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerName="cleanup"
Nov 01 04:15:20.159403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159395    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerName="cleanup"
Nov 01 04:15:20.159953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159411    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38030cd-cbfc-4729-b270-e456184ec6d4" containerName="revoke-keys"
Nov 01 04:15:20.159953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159423    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerName="cleanup"
Nov 01 04:15:20.159953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159451    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerName="cleanup"
Nov 01 04:15:20.159953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159463    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerName="cleanup"
Nov 01 04:15:20.159953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159473    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818" containerName="cleanup"
Nov 01 04:15:20.159953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159481    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerName="cleanup"
Nov 01 04:15:20.159953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.159488    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerName="cleanup"
Nov 01 04:15:20.163307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.163287    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:20.214455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.214402    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:15:20.216884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.216857    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"]
Nov 01 04:15:20.244172 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.244143    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"]
Nov 01 04:15:20.249076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.248908    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732324a5-5bcnf\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:20.249076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.248962    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732324a5-5bcnf\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:20.249076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.248986    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-157fed-29891775-rctd9"]
Nov 01 04:15:20.249308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.249215    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dprf8\" (UniqueName: \"kubernetes.io/projected/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-kube-api-access-dprf8\") pod \"maas-api-revoke-sub-732324a5-5bcnf\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:20.350738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.350683    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprf8\" (UniqueName: \"kubernetes.io/projected/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-kube-api-access-dprf8\") pod \"maas-api-revoke-sub-732324a5-5bcnf\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:20.350945 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.350792    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732324a5-5bcnf\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:20.350945 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.350813    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732324a5-5bcnf\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:20.351513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.351485    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732324a5-5bcnf\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:20.353394 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.353368    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732324a5-5bcnf\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:20.411482 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.411387    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dprf8\" (UniqueName: \"kubernetes.io/projected/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-kube-api-access-dprf8\") pod \"maas-api-revoke-sub-732324a5-5bcnf\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:20.476569 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.476529    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:20.624403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.624366    4882 generic.go:356] "Generic (PLEG): container finished" podID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerID="534b2b60789ba981d371645f607182e8ed76e889906a02606c1611e4081d2365" exitCode=60
Nov 01 04:15:20.624588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.624400    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" event={"ID":"6ff2266b-1617-4268-b5e4-fe0d0c57b352","Type":"ContainerDied","Data":"534b2b60789ba981d371645f607182e8ed76e889906a02606c1611e4081d2365"}
Nov 01 04:15:20.624588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.624494    4882 scope.go:122] "RemoveContainer" containerID="d0edf66f644a407758daceda6d208a361a2c46a3aa2fa74c6a0486f58a118e9d"
Nov 01 04:15:20.625035 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.625012    4882 scope.go:122] "RemoveContainer" containerID="534b2b60789ba981d371645f607182e8ed76e889906a02606c1611e4081d2365"
Nov 01 04:15:20.625464 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:20.625402    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx_odh-ai-gateway-infra(6ff2266b-1617-4268-b5e4-fe0d0c57b352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352"
Nov 01 04:15:20.628792 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.628769    4882 generic.go:356] "Generic (PLEG): container finished" podID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerID="6c0c37e14b744883973856367d78df36deed4e7abca8189421f9f04d8d5bfc0d" exitCode=60
Nov 01 04:15:20.628867 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.628843    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" event={"ID":"4d774ed2-7110-4cd5-b8e2-20de7406e20b","Type":"ContainerDied","Data":"6c0c37e14b744883973856367d78df36deed4e7abca8189421f9f04d8d5bfc0d"}
Nov 01 04:15:20.629298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.629280    4882 scope.go:122] "RemoveContainer" containerID="6c0c37e14b744883973856367d78df36deed4e7abca8189421f9f04d8d5bfc0d"
Nov 01 04:15:20.629740 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:20.629715    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb_odh-ai-gateway-infra(4d774ed2-7110-4cd5-b8e2-20de7406e20b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b"
Nov 01 04:15:20.630865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.630848    4882 generic.go:356] "Generic (PLEG): container finished" podID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerID="9e8675ccb1668fdbd02b3782db6aab317c870ccd860c3f72dbf2b891f01ea138" exitCode=60
Nov 01 04:15:20.630951 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.630924    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt" event={"ID":"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1","Type":"ContainerDied","Data":"9e8675ccb1668fdbd02b3782db6aab317c870ccd860c3f72dbf2b891f01ea138"}
Nov 01 04:15:20.631323 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.631307    4882 scope.go:122] "RemoveContainer" containerID="9e8675ccb1668fdbd02b3782db6aab317c870ccd860c3f72dbf2b891f01ea138"
Nov 01 04:15:20.631689 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:20.631653    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt_odh-ai-gateway-infra(0c2fb0aa-2c54-489a-a22d-e2d25f014ee1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1"
Nov 01 04:15:20.632452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.632401    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"
Nov 01 04:15:20.632452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.632414    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6" event={"ID":"290b65f4-af57-4e95-b59e-414f4d2d9480","Type":"ContainerDied","Data":"f1e5fe19030bb822cf40e69262394bff09a822a51633da651d3a4896f2c1c962"}
Nov 01 04:15:20.633983 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.633949    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp" event={"ID":"bc5e01fe-46aa-4663-8788-c50f05c2d818","Type":"ContainerDied","Data":"a24008e5b84a08253940b5e9cf485d690454523d8f7261bf731a145f71fb78e7"}
Nov 01 04:15:20.634143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.633953    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"
Nov 01 04:15:20.635974 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.635956    4882 generic.go:356] "Generic (PLEG): container finished" podID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerID="de2a2a72481f12d76a0687d8320f4c594c94e9c78a01c0bd87426f2f9553510d" exitCode=60
Nov 01 04:15:20.636059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.635986    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" event={"ID":"cad53e56-0eb2-49f3-8fea-7cea9e035732","Type":"ContainerDied","Data":"de2a2a72481f12d76a0687d8320f4c594c94e9c78a01c0bd87426f2f9553510d"}
Nov 01 04:15:20.636263 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.636250    4882 scope.go:122] "RemoveContainer" containerID="de2a2a72481f12d76a0687d8320f4c594c94e9c78a01c0bd87426f2f9553510d"
Nov 01 04:15:20.636487 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:20.636467    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891775-dm9xs_odh-ai-gateway-infra(cad53e56-0eb2-49f3-8fea-7cea9e035732)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732"
Nov 01 04:15:20.650937 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.650916    4882 scope.go:122] "RemoveContainer" containerID="d1bb5df46040e001eb4bf8c418ce3d91ddb455eec161dc22b4b80ab5dcd6cebf"
Nov 01 04:15:20.663988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.663968    4882 scope.go:122] "RemoveContainer" containerID="adf28bce4677bb11491a6466d28c8c50a254f61a52460bd747a8f5487e058459"
Nov 01 04:15:20.677006 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.676986    4882 scope.go:122] "RemoveContainer" containerID="772bc78184d55c1a3de7e2fb82df651b8590283dac35348fed88e9f53894b198"
Nov 01 04:15:20.685751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.685723    4882 scope.go:122] "RemoveContainer" containerID="ad81f9411a19e9ea6557546357dc1ce582467c214a2ae3bde2a2c1a752775450"
Nov 01 04:15:20.694406 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.694387    4882 scope.go:122] "RemoveContainer" containerID="71e114fe89762fe3990449864315f408c36ebedb4eb5e88e9bac022264187a7e"
Nov 01 04:15:20.830477 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.830438    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"]
Nov 01 04:15:20.844875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.844845    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"]
Nov 01 04:15:20.853411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.853314    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"]
Nov 01 04:15:20.854639 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:15:20.854608    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093dd5d0_8f3a_4d24_9800_af71c61a4b3b.slice/crio-18711d4bf3ca50e70d4af26268f1a49c0244f2a368383d3cca7654942ab6b7f7 WatchSource:0}: Error finding container 18711d4bf3ca50e70d4af26268f1a49c0244f2a368383d3cca7654942ab6b7f7: Status 404 returned error can't find the container with id 18711d4bf3ca50e70d4af26268f1a49c0244f2a368383d3cca7654942ab6b7f7
Nov 01 04:15:20.858187 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.858117    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-535e84-29891775-9nlq6"]
Nov 01 04:15:20.935705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:20.935677    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"]
Nov 01 04:15:21.042421 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.042384    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"]
Nov 01 04:15:21.150224 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.150186    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"]
Nov 01 04:15:21.284252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.284164    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"]
Nov 01 04:15:21.289158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.289127    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f73b9f-29891775-t6jvp"]
Nov 01 04:15:21.607723 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.607640    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480" path="/var/lib/kubelet/pods/290b65f4-af57-4e95-b59e-414f4d2d9480/volumes"
Nov 01 04:15:21.608102 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.608088    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc5e01fe-46aa-4663-8788-c50f05c2d818" path="/var/lib/kubelet/pods/bc5e01fe-46aa-4663-8788-c50f05c2d818/volumes"
Nov 01 04:15:21.608488 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.608475    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671" path="/var/lib/kubelet/pods/da64a22b-d936-4d72-aac2-d3d9f8432671/volumes"
Nov 01 04:15:21.649268 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.649236    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf" event={"ID":"093dd5d0-8f3a-4d24-9800-af71c61a4b3b","Type":"ContainerStarted","Data":"359b78135246edae54f29038f8d8bd0f16c74b6f57baefef71d4fc4acbbb692b"}
Nov 01 04:15:21.649391 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.649299    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf" event={"ID":"093dd5d0-8f3a-4d24-9800-af71c61a4b3b","Type":"ContainerStarted","Data":"18711d4bf3ca50e70d4af26268f1a49c0244f2a368383d3cca7654942ab6b7f7"}
Nov 01 04:15:21.832021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.831990    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"
Nov 01 04:15:21.871414 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.871328    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d774ed2-7110-4cd5-b8e2-20de7406e20b-kube-api-access-p6rng\" (UniqueName: \"kubernetes.io/projected/4d774ed2-7110-4cd5-b8e2-20de7406e20b-kube-api-access-p6rng\") pod \"4d774ed2-7110-4cd5-b8e2-20de7406e20b\" (UID: \"4d774ed2-7110-4cd5-b8e2-20de7406e20b\") "
Nov 01 04:15:21.871414 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.871391    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d774ed2-7110-4cd5-b8e2-20de7406e20b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d774ed2-7110-4cd5-b8e2-20de7406e20b-maas-api-service-ca\") pod \"4d774ed2-7110-4cd5-b8e2-20de7406e20b\" (UID: \"4d774ed2-7110-4cd5-b8e2-20de7406e20b\") "
Nov 01 04:15:21.873161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.872755    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d774ed2-7110-4cd5-b8e2-20de7406e20b-maas-api-service-ca" pod "4d774ed2-7110-4cd5-b8e2-20de7406e20b" (UID: "4d774ed2-7110-4cd5-b8e2-20de7406e20b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:21.877026 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.876996    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d774ed2-7110-4cd5-b8e2-20de7406e20b-kube-api-access-p6rng" pod "4d774ed2-7110-4cd5-b8e2-20de7406e20b" (UID: "4d774ed2-7110-4cd5-b8e2-20de7406e20b"). InnerVolumeSpecName "kube-api-access-p6rng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:21.900125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.900101    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"
Nov 01 04:15:21.937846 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.937818    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"
Nov 01 04:15:21.941409 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.941390    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"
Nov 01 04:15:21.972627 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.972594    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff2266b-1617-4268-b5e4-fe0d0c57b352-kube-api-access-gnrt6\" (UniqueName: \"kubernetes.io/projected/6ff2266b-1617-4268-b5e4-fe0d0c57b352-kube-api-access-gnrt6\") pod \"6ff2266b-1617-4268-b5e4-fe0d0c57b352\" (UID: \"6ff2266b-1617-4268-b5e4-fe0d0c57b352\") "
Nov 01 04:15:21.972800 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.972658    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-maas-api-service-ca\") pod \"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1\" (UID: \"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1\") "
Nov 01 04:15:21.972800 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.972713    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad53e56-0eb2-49f3-8fea-7cea9e035732-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad53e56-0eb2-49f3-8fea-7cea9e035732-maas-api-service-ca\") pod \"cad53e56-0eb2-49f3-8fea-7cea9e035732\" (UID: \"cad53e56-0eb2-49f3-8fea-7cea9e035732\") "
Nov 01 04:15:21.972800 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.972772    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad53e56-0eb2-49f3-8fea-7cea9e035732-kube-api-access-pbkmr\" (UniqueName: \"kubernetes.io/projected/cad53e56-0eb2-49f3-8fea-7cea9e035732-kube-api-access-pbkmr\") pod \"cad53e56-0eb2-49f3-8fea-7cea9e035732\" (UID: \"cad53e56-0eb2-49f3-8fea-7cea9e035732\") "
Nov 01 04:15:21.973084 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.972805    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff2266b-1617-4268-b5e4-fe0d0c57b352-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff2266b-1617-4268-b5e4-fe0d0c57b352-maas-api-service-ca\") pod \"6ff2266b-1617-4268-b5e4-fe0d0c57b352\" (UID: \"6ff2266b-1617-4268-b5e4-fe0d0c57b352\") "
Nov 01 04:15:21.973084 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.972867    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-kube-api-access-pqxxh\" (UniqueName: \"kubernetes.io/projected/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-kube-api-access-pqxxh\") pod \"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1\" (UID: \"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1\") "
Nov 01 04:15:21.973251 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.973197    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad53e56-0eb2-49f3-8fea-7cea9e035732-maas-api-service-ca" pod "cad53e56-0eb2-49f3-8fea-7cea9e035732" (UID: "cad53e56-0eb2-49f3-8fea-7cea9e035732"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:21.973548 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.973514    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff2266b-1617-4268-b5e4-fe0d0c57b352-maas-api-service-ca" pod "6ff2266b-1617-4268-b5e4-fe0d0c57b352" (UID: "6ff2266b-1617-4268-b5e4-fe0d0c57b352"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:21.973737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.973687    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6rng\" (UniqueName: \"kubernetes.io/projected/4d774ed2-7110-4cd5-b8e2-20de7406e20b-kube-api-access-p6rng\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:21.973737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.973711    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d774ed2-7110-4cd5-b8e2-20de7406e20b-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:21.973737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.973727    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad53e56-0eb2-49f3-8fea-7cea9e035732-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:21.973910 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.973741    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff2266b-1617-4268-b5e4-fe0d0c57b352-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:21.973910 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.973737    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-maas-api-service-ca" pod "0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" (UID: "0c2fb0aa-2c54-489a-a22d-e2d25f014ee1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:21.975332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.975311    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff2266b-1617-4268-b5e4-fe0d0c57b352-kube-api-access-gnrt6" pod "6ff2266b-1617-4268-b5e4-fe0d0c57b352" (UID: "6ff2266b-1617-4268-b5e4-fe0d0c57b352"). InnerVolumeSpecName "kube-api-access-gnrt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:21.975483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.975345    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-kube-api-access-pqxxh" pod "0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" (UID: "0c2fb0aa-2c54-489a-a22d-e2d25f014ee1"). InnerVolumeSpecName "kube-api-access-pqxxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:21.976205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.976178    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad53e56-0eb2-49f3-8fea-7cea9e035732-kube-api-access-pbkmr" pod "cad53e56-0eb2-49f3-8fea-7cea9e035732" (UID: "cad53e56-0eb2-49f3-8fea-7cea9e035732"). InnerVolumeSpecName "kube-api-access-pbkmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:21.976660 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:21.976619    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf" podStartSLOduration=1.976606386 podStartE2EDuration="1.976606386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:21.972000063 +0000 UTC m=+1990.534033080" watchObservedRunningTime="2026-11-01 04:15:21.976606386 +0000 UTC m=+1990.538639402"
Nov 01 04:15:22.075417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.075377    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnrt6\" (UniqueName: \"kubernetes.io/projected/6ff2266b-1617-4268-b5e4-fe0d0c57b352-kube-api-access-gnrt6\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:22.075417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.075407    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:22.075417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.075417    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbkmr\" (UniqueName: \"kubernetes.io/projected/cad53e56-0eb2-49f3-8fea-7cea9e035732-kube-api-access-pbkmr\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:22.075698 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.075443    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqxxh\" (UniqueName: \"kubernetes.io/projected/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1-kube-api-access-pqxxh\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:22.663700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.663662    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"
Nov 01 04:15:22.663700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.663687    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb" event={"ID":"4d774ed2-7110-4cd5-b8e2-20de7406e20b","Type":"ContainerDied","Data":"b740eb958b77b0067e295ff67622389b544f4708b67d615d029a18e4d71fbf5b"}
Nov 01 04:15:22.663920 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.663755    4882 scope.go:122] "RemoveContainer" containerID="6c0c37e14b744883973856367d78df36deed4e7abca8189421f9f04d8d5bfc0d"
Nov 01 04:15:22.665348 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.665325    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt" event={"ID":"0c2fb0aa-2c54-489a-a22d-e2d25f014ee1","Type":"ContainerDied","Data":"47a8e4c5ca724f6081b44a1f5f1c2fdf779fb449f18673ce930eabd1b96e1ff9"}
Nov 01 04:15:22.665483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.665405    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"
Nov 01 04:15:22.667710 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.667683    4882 generic.go:356] "Generic (PLEG): container finished" podID="093dd5d0-8f3a-4d24-9800-af71c61a4b3b" containerID="359b78135246edae54f29038f8d8bd0f16c74b6f57baefef71d4fc4acbbb692b" exitCode=0
Nov 01 04:15:22.667840 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.667789    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf" event={"ID":"093dd5d0-8f3a-4d24-9800-af71c61a4b3b","Type":"ContainerDied","Data":"359b78135246edae54f29038f8d8bd0f16c74b6f57baefef71d4fc4acbbb692b"}
Nov 01 04:15:22.669263 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.669236    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs" event={"ID":"cad53e56-0eb2-49f3-8fea-7cea9e035732","Type":"ContainerDied","Data":"e9decec73418dff06cd150bff93685b1ae405eb6a7504a423ef70150d91c08d7"}
Nov 01 04:15:22.669357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.669335    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"
Nov 01 04:15:22.671006 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.670980    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx" event={"ID":"6ff2266b-1617-4268-b5e4-fe0d0c57b352","Type":"ContainerDied","Data":"73b4713b55ae4d0d8820a23bd6b2cfb6e5ba00390393b7251f07648865ea5f4f"}
Nov 01 04:15:22.671145 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.670993    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"
Nov 01 04:15:22.677073 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.677051    4882 scope.go:122] "RemoveContainer" containerID="9e8675ccb1668fdbd02b3782db6aab317c870ccd860c3f72dbf2b891f01ea138"
Nov 01 04:15:22.688085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.688065    4882 scope.go:122] "RemoveContainer" containerID="de2a2a72481f12d76a0687d8320f4c594c94e9c78a01c0bd87426f2f9553510d"
Nov 01 04:15:22.698621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.698600    4882 scope.go:122] "RemoveContainer" containerID="534b2b60789ba981d371645f607182e8ed76e889906a02606c1611e4081d2365"
Nov 01 04:15:22.818016 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.817971    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"]
Nov 01 04:15:22.821473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.821445    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-dm9xs"]
Nov 01 04:15:22.922904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.922820    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"]
Nov 01 04:15:22.926466 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:22.926437    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891775-s94sb"]
Nov 01 04:15:23.027738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.027705    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"]
Nov 01 04:15:23.032025 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.031999    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891775-7tpvt"]
Nov 01 04:15:23.132736 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.132699    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"]
Nov 01 04:15:23.136927 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.136900    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5b71cd-29891775-mstpx"]
Nov 01 04:15:23.602968 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.602931    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" path="/var/lib/kubelet/pods/0c2fb0aa-2c54-489a-a22d-e2d25f014ee1/volumes"
Nov 01 04:15:23.603889 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.603860    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" path="/var/lib/kubelet/pods/4d774ed2-7110-4cd5-b8e2-20de7406e20b/volumes"
Nov 01 04:15:23.604692 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.604671    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" path="/var/lib/kubelet/pods/6ff2266b-1617-4268-b5e4-fe0d0c57b352/volumes"
Nov 01 04:15:23.605908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.605889    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732" path="/var/lib/kubelet/pods/cad53e56-0eb2-49f3-8fea-7cea9e035732/volumes"
Nov 01 04:15:23.856579 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.856513    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:23.898210 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.898162    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-xks-ca\") pod \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") "
Nov 01 04:15:23.898395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.898307    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-openshift-ca\") pod \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") "
Nov 01 04:15:23.898465 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.898406    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-kube-api-access-dprf8\" (UniqueName: \"kubernetes.io/projected/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-kube-api-access-dprf8\") pod \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\" (UID: \"093dd5d0-8f3a-4d24-9800-af71c61a4b3b\") "
Nov 01 04:15:23.898680 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.898648    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-openshift-ca" pod "093dd5d0-8f3a-4d24-9800-af71c61a4b3b" (UID: "093dd5d0-8f3a-4d24-9800-af71c61a4b3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:23.898876 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.898861    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:23.900406 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.900382    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-kube-api-access-dprf8" pod "093dd5d0-8f3a-4d24-9800-af71c61a4b3b" (UID: "093dd5d0-8f3a-4d24-9800-af71c61a4b3b"). InnerVolumeSpecName "kube-api-access-dprf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:23.900560 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:23.900403    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-xks-ca" pod "093dd5d0-8f3a-4d24-9800-af71c61a4b3b" (UID: "093dd5d0-8f3a-4d24-9800-af71c61a4b3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:15:24.000882 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:24.000841    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:24.000882 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:24.000881    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dprf8\" (UniqueName: \"kubernetes.io/projected/093dd5d0-8f3a-4d24-9800-af71c61a4b3b-kube-api-access-dprf8\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:24.693286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:24.693251    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"
Nov 01 04:15:24.693541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:24.693291    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf" event={"ID":"093dd5d0-8f3a-4d24-9800-af71c61a4b3b","Type":"ContainerDied","Data":"18711d4bf3ca50e70d4af26268f1a49c0244f2a368383d3cca7654942ab6b7f7"}
Nov 01 04:15:24.693541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:24.693338    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18711d4bf3ca50e70d4af26268f1a49c0244f2a368383d3cca7654942ab6b7f7"
Nov 01 04:15:28.725852 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:28.725807    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:15:34.175947 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:34.175911    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"]
Nov 01 04:15:43.147990 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.147952    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"]
Nov 01 04:15:43.148552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148533    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerName="cleanup"
Nov 01 04:15:43.148633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148553    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerName="cleanup"
Nov 01 04:15:43.148633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148571    4882 cpu_manager.go:414] "containerMap: removing container" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerName="cleanup"
Nov 01 04:15:43.148633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148576    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290b65f4-af57-4e95-b59e-414f4d2d9480" containerName="cleanup"
Nov 01 04:15:43.148633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148583    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerName="cleanup"
Nov 01 04:15:43.148633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148587    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerName="cleanup"
Nov 01 04:15:43.148633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148600    4882 cpu_manager.go:414] "containerMap: removing container" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerName="cleanup"
Nov 01 04:15:43.148633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148604    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da64a22b-d936-4d72-aac2-d3d9f8432671" containerName="cleanup"
Nov 01 04:15:43.148633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148621    4882 cpu_manager.go:414] "containerMap: removing container" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerName="cleanup"
Nov 01 04:15:43.148633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148626    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerName="cleanup"
Nov 01 04:15:43.148633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148639    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148644    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148653    4882 cpu_manager.go:414] "containerMap: removing container" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148657    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148671    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148678    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148694    4882 cpu_manager.go:414] "containerMap: removing container" podUID="093dd5d0-8f3a-4d24-9800-af71c61a4b3b" containerName="revoke-keys"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148700    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093dd5d0-8f3a-4d24-9800-af71c61a4b3b" containerName="revoke-keys"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148708    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148713    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148719    4882 cpu_manager.go:414] "containerMap: removing container" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148722    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148728    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148732    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148744    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148750    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148843    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148852    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148858    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148863    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148870    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148879    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148885    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e51db6-b4d2-4f75-99b9-1fd0a9c5dfef" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148891    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148898    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerName="cleanup"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148909    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="093dd5d0-8f3a-4d24-9800-af71c61a4b3b" containerName="revoke-keys"
Nov 01 04:15:43.149068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.148916    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerName="cleanup"
Nov 01 04:15:43.152616 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.152596    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.210861 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.210823    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3a348f-openshift-default-dockercfg-fmnm5\""
Nov 01 04:15:43.251453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.251267    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qbf\" (UniqueName: \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-kube-api-access-m5qbf\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.251453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.251356    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-token\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.251453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.251386    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-data\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.251784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.251544    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istiod-ca-cert\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.251784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.251607    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-podinfo\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.251784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.251632    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-socket\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.251784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.251699    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-envoy\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.251784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.251758    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-certs\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.252064 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.251812    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-credential-socket\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.292455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.266389    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"]
Nov 01 04:15:43.353864 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.353812    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-envoy\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.354188 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.354160    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-certs\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.354369 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.354350    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-credential-socket\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.354611 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.354591    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qbf\" (UniqueName: \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-kube-api-access-m5qbf\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.354759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.354744    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-token\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.354864 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.354849    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-data\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.355054 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.355035    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istiod-ca-cert\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.355234 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.355217    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-podinfo\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.355373 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.355353    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-socket\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.355796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.355771    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-socket\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.356232 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.356209    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-certs\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.356674 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.356655    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-credential-socket\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.357621 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.357599    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-data\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.358397 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.358376    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istiod-ca-cert\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.401032 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.379592    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-podinfo\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.401032 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.385936    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-envoy\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.418020 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.417961    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qbf\" (UniqueName: \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-kube-api-access-m5qbf\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.419159 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.419111    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-token\") pod \"e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") " pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.466277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.466240    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:43.922941 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.922905    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"]
Nov 01 04:15:43.924753 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:15:43.924725    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5804f1e8_7bd2_443d_89ef_8bb1389845d1.slice/crio-5c8a49d6e174cd25b397fcd85c1e419dff3e6813b9408bf19a6d0174aea8a916 WatchSource:0}: Error finding container 5c8a49d6e174cd25b397fcd85c1e419dff3e6813b9408bf19a6d0174aea8a916: Status 404 returned error can't find the container with id 5c8a49d6e174cd25b397fcd85c1e419dff3e6813b9408bf19a6d0174aea8a916
Nov 01 04:15:43.927772 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.927724    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:15:43.927941 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.927830    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:15:43.927941 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:43.927883    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:15:44.848562 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:44.848520    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" event={"ID":"5804f1e8-7bd2-443d-89ef-8bb1389845d1","Type":"ContainerStarted","Data":"516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b"}
Nov 01 04:15:44.848975 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:44.848613    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" event={"ID":"5804f1e8-7bd2-443d-89ef-8bb1389845d1","Type":"ContainerStarted","Data":"5c8a49d6e174cd25b397fcd85c1e419dff3e6813b9408bf19a6d0174aea8a916"}
Nov 01 04:15:45.008706 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:45.008648    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" podStartSLOduration=2.008634933 podStartE2EDuration="2.008634933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:45.007561561 +0000 UTC m=+2013.569594577" watchObservedRunningTime="2026-11-01 04:15:45.008634933 +0000 UTC m=+2013.570667946"
Nov 01 04:15:45.466556 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:45.466524    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:45.472821 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:45.472791    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:45.855409 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:45.855326    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:45.856535 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:45.856512    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:15:49.196757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.196715    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj" podUID="ea856e65-e162-4031-ac33-acf56ce4b296" containerName="main" containerID="cri-o://90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e" gracePeriod=45
Nov 01 04:15:49.629579 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.629549    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:15:49.815761 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.815724    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-home\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-home\") pod \"ea856e65-e162-4031-ac33-acf56ce4b296\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") "
Nov 01 04:15:49.815970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.815767    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-model-cache\") pod \"ea856e65-e162-4031-ac33-acf56ce4b296\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") "
Nov 01 04:15:49.815970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.815868    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea856e65-e162-4031-ac33-acf56ce4b296-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea856e65-e162-4031-ac33-acf56ce4b296-tls-certs\") pod \"ea856e65-e162-4031-ac33-acf56ce4b296\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") "
Nov 01 04:15:49.815970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.815932    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-dshm\") pod \"ea856e65-e162-4031-ac33-acf56ce4b296\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") "
Nov 01 04:15:49.816138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.816021    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-home" pod "ea856e65-e162-4031-ac33-acf56ce4b296" (UID: "ea856e65-e162-4031-ac33-acf56ce4b296"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:15:49.816138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.816058    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea856e65-e162-4031-ac33-acf56ce4b296-kube-api-access-wg626\" (UniqueName: \"kubernetes.io/projected/ea856e65-e162-4031-ac33-acf56ce4b296-kube-api-access-wg626\") pod \"ea856e65-e162-4031-ac33-acf56ce4b296\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") "
Nov 01 04:15:49.816138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.816054    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-model-cache" pod "ea856e65-e162-4031-ac33-acf56ce4b296" (UID: "ea856e65-e162-4031-ac33-acf56ce4b296"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:15:49.816138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.816125    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-tmp-dir\") pod \"ea856e65-e162-4031-ac33-acf56ce4b296\" (UID: \"ea856e65-e162-4031-ac33-acf56ce4b296\") "
Nov 01 04:15:49.816348 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.816327    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-tmp-dir" pod "ea856e65-e162-4031-ac33-acf56ce4b296" (UID: "ea856e65-e162-4031-ac33-acf56ce4b296"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:15:49.816773 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.816746    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-tmp-dir\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:49.816773 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.816773    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-home\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:49.817006 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.816786    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-model-cache\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:49.818258 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.818231    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-dshm" pod "ea856e65-e162-4031-ac33-acf56ce4b296" (UID: "ea856e65-e162-4031-ac33-acf56ce4b296"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:15:49.818362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.818298    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea856e65-e162-4031-ac33-acf56ce4b296-tls-certs" pod "ea856e65-e162-4031-ac33-acf56ce4b296" (UID: "ea856e65-e162-4031-ac33-acf56ce4b296"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:15:49.818362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.818309    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea856e65-e162-4031-ac33-acf56ce4b296-kube-api-access-wg626" pod "ea856e65-e162-4031-ac33-acf56ce4b296" (UID: "ea856e65-e162-4031-ac33-acf56ce4b296"). InnerVolumeSpecName "kube-api-access-wg626". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:49.884639 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.884553    4882 generic.go:356] "Generic (PLEG): container finished" podID="ea856e65-e162-4031-ac33-acf56ce4b296" containerID="90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e" exitCode=0
Nov 01 04:15:49.884639 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.884595    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"
Nov 01 04:15:49.884863 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.884635    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj" event={"ID":"ea856e65-e162-4031-ac33-acf56ce4b296","Type":"ContainerDied","Data":"90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e"}
Nov 01 04:15:49.884863 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.884677    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj" event={"ID":"ea856e65-e162-4031-ac33-acf56ce4b296","Type":"ContainerDied","Data":"8f1eb554a9f40a7174dbbf09abeb8a034fb9cfc4ed3959b3d677bade539ada72"}
Nov 01 04:15:49.884863 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.884694    4882 scope.go:122] "RemoveContainer" containerID="90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e"
Nov 01 04:15:49.896416 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.896390    4882 scope.go:122] "RemoveContainer" containerID="90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e"
Nov 01 04:15:49.896737 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:49.896715    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e\": container with ID starting with 90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e not found: ID does not exist" containerID="90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e"
Nov 01 04:15:49.896795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.896745    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e"} err="failed to get container status \"90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e\": rpc error: code = NotFound desc = could not find container \"90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e\": container with ID starting with 90f35faf0a4da481c71360071155bf6b2c5bfb7869cbcebc03add961980eda7e not found: ID does not exist"
Nov 01 04:15:49.918660 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.918633    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea856e65-e162-4031-ac33-acf56ce4b296-tls-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:49.918660 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.918659    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea856e65-e162-4031-ac33-acf56ce4b296-dshm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:49.918865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:49.918670    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg626\" (UniqueName: \"kubernetes.io/projected/ea856e65-e162-4031-ac33-acf56ce4b296-kube-api-access-wg626\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:15:50.038551 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:50.038484    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"]
Nov 01 04:15:50.042454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:50.042413    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4adf4c/tenantcfg-model-4adf4c-kserve-5b98864449-bnzhj"]
Nov 01 04:15:51.687418 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:15:51.687318    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea856e65-e162-4031-ac33-acf56ce4b296" path="/var/lib/kubelet/pods/ea856e65-e162-4031-ac33-acf56ce4b296/volumes"
Nov 01 04:15:58.724585 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:15:58.724543    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:16:05.825829 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.825780    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"]
Nov 01 04:16:05.826462 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.826438    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ea856e65-e162-4031-ac33-acf56ce4b296" containerName="main"
Nov 01 04:16:05.826462 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.826459    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea856e65-e162-4031-ac33-acf56ce4b296" containerName="main"
Nov 01 04:16:05.826585 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.826484    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerName="cleanup"
Nov 01 04:16:05.826585 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.826489    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff2266b-1617-4268-b5e4-fe0d0c57b352" containerName="cleanup"
Nov 01 04:16:05.826585 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.826513    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerName="cleanup"
Nov 01 04:16:05.826585 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.826519    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerName="cleanup"
Nov 01 04:16:05.826802 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.826614    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea856e65-e162-4031-ac33-acf56ce4b296" containerName="main"
Nov 01 04:16:05.826802 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.826628    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad53e56-0eb2-49f3-8fea-7cea9e035732" containerName="cleanup"
Nov 01 04:16:05.826802 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.826637    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d774ed2-7110-4cd5-b8e2-20de7406e20b" containerName="cleanup"
Nov 01 04:16:05.826802 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.826644    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2fb0aa-2c54-489a-a22d-e2d25f014ee1" containerName="cleanup"
Nov 01 04:16:05.830539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.830516    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:05.879383 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.879341    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-601a4b-openshift-default-dockercfg-ccdnd\""
Nov 01 04:16:05.931566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.931530    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"]
Nov 01 04:16:05.959874 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.959836    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xshzs\" (UniqueName: \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-kube-api-access-xshzs\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:05.960035 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.959891    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-certs\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:05.960105 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.960063    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-envoy\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:05.960226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.960186    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-token\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:05.960289 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.960232    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-credential-socket\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:05.960414 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.960392    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-podinfo\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:05.960519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.960500    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istiod-ca-cert\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:05.960581 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.960547    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-socket\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:05.960709 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:05.960686    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-data\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.064601 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.064545    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-podinfo\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.064918 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.064875    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istiod-ca-cert\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.065143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.065108    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-socket\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.065468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.065447    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-data\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.065640 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.065617    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xshzs\" (UniqueName: \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-kube-api-access-xshzs\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.065771 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.065757    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-certs\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.066068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.066049    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-envoy\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.068795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.067909    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istiod-ca-cert\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.068795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.068498    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-data\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.068795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.068749    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-certs\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.070942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.070919    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-token\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.077614 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.071177    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-credential-socket\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.077850 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.077527    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-credential-socket\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.078739 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.078707    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-socket\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.082065 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.082012    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-envoy\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.103491 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.097098    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-podinfo\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.136741 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.136700    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-token\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.136907 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.136776    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xshzs\" (UniqueName: \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-kube-api-access-xshzs\") pod \"e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") " pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.145711 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.145681    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:06.569726 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.569691    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"]
Nov 01 04:16:06.603531 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:16:06.603502    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769e10a8_d82e_4a7c_aec7_b79eb2cf1223.slice/crio-b724052bd4952b909e0e85d2da18f1b2fa4a2649945c2ce6acfdf3b804541a5b WatchSource:0}: Error finding container b724052bd4952b909e0e85d2da18f1b2fa4a2649945c2ce6acfdf3b804541a5b: Status 404 returned error can't find the container with id b724052bd4952b909e0e85d2da18f1b2fa4a2649945c2ce6acfdf3b804541a5b
Nov 01 04:16:06.606098 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.606063    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:16:06.606196 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.606140    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:16:06.606196 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:06.606169    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:16:07.050269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:07.050220    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" event={"ID":"769e10a8-d82e-4a7c-aec7-b79eb2cf1223","Type":"ContainerStarted","Data":"03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827"}
Nov 01 04:16:07.050269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:07.050274    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" event={"ID":"769e10a8-d82e-4a7c-aec7-b79eb2cf1223","Type":"ContainerStarted","Data":"b724052bd4952b909e0e85d2da18f1b2fa4a2649945c2ce6acfdf3b804541a5b"}
Nov 01 04:16:07.145974 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:07.145933    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:07.151839 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:07.151810    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:07.165413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:07.165364    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" podStartSLOduration=2.165349296 podStartE2EDuration="2.165349296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:16:07.162693089 +0000 UTC m=+2035.724726104" watchObservedRunningTime="2026-11-01 04:16:07.165349296 +0000 UTC m=+2035.727382406"
Nov 01 04:16:08.056926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:08.056892    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:08.057984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:08.057960    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:16:09.704903 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:09.704864    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"]
Nov 01 04:16:09.705312 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:09.705086    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy" containerID="cri-o://f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b" gracePeriod=30
Nov 01 04:16:13.131990 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:13.131952    4882 scope.go:122] "RemoveContainer" containerID="6d87962f98a5fc82576e9bae1384fc291dfb93ebf335f03ed05b9a2c1e249ccb"
Nov 01 04:16:13.145134 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:13.145111    4882 scope.go:122] "RemoveContainer" containerID="2b7bd92051b910c5fcc7260e2e3c262565662ada44b21d951766956b71ad7541"
Nov 01 04:16:15.048866 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.048800    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 04:16:15.106684 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.106646    4882 generic.go:356] "Generic (PLEG): container finished" podID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerID="f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b" exitCode=0
Nov 01 04:16:15.106869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.106706    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"
Nov 01 04:16:15.106869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.106696    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" event={"ID":"81be89f4-ebde-4d7b-8e24-18bd4a385824","Type":"ContainerDied","Data":"f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b"}
Nov 01 04:16:15.106869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.106833    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x" event={"ID":"81be89f4-ebde-4d7b-8e24-18bd4a385824","Type":"ContainerDied","Data":"4d68925f6318c4527d83ff65df38a42967d099c03a20432964e67b8836ecfcd2"}
Nov 01 04:16:15.106869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.106858    4882 scope.go:122] "RemoveContainer" containerID="f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b"
Nov 01 04:16:15.118005 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.117980    4882 scope.go:122] "RemoveContainer" containerID="ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8"
Nov 01 04:16:15.127493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.127468    4882 scope.go:122] "RemoveContainer" containerID="f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b"
Nov 01 04:16:15.127775 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:16:15.127752    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b\": container with ID starting with f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b not found: ID does not exist" containerID="f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b"
Nov 01 04:16:15.127880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.127789    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b"} err="failed to get container status \"f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b\": rpc error: code = NotFound desc = could not find container \"f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b\": container with ID starting with f5518d56a9356f6c992d7a3f0e552cb4b86873b43d07c23bafd0e1bc8078518b not found: ID does not exist"
Nov 01 04:16:15.127880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.127815    4882 scope.go:122] "RemoveContainer" containerID="ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8"
Nov 01 04:16:15.128118 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:16:15.128097    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8\": container with ID starting with ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8 not found: ID does not exist" containerID="ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8"
Nov 01 04:16:15.128160 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.128127    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8"} err="failed to get container status \"ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8\": rpc error: code = NotFound desc = could not find container \"ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8\": container with ID starting with ba7d1757f71c63dc6decbb1e5afa2a4848a7c161fcbf7ff23fb448e0867a83b8 not found: ID does not exist"
Nov 01 04:16:15.148825 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.148803    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-envoy\") pod \"81be89f4-ebde-4d7b-8e24-18bd4a385824\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") "
Nov 01 04:16:15.148946 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.148892    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-podinfo\") pod \"81be89f4-ebde-4d7b-8e24-18bd4a385824\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") "
Nov 01 04:16:15.148946 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.148930    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-socket\") pod \"81be89f4-ebde-4d7b-8e24-18bd4a385824\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") "
Nov 01 04:16:15.149017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.148979    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-token\" (UniqueName: \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-token\") pod \"81be89f4-ebde-4d7b-8e24-18bd4a385824\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") "
Nov 01 04:16:15.149060 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.149013    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-data\") pod \"81be89f4-ebde-4d7b-8e24-18bd4a385824\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") "
Nov 01 04:16:15.149174 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.149151    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-credential-socket\") pod \"81be89f4-ebde-4d7b-8e24-18bd4a385824\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") "
Nov 01 04:16:15.149314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.149186    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-socket" pod "81be89f4-ebde-4d7b-8e24-18bd4a385824" (UID: "81be89f4-ebde-4d7b-8e24-18bd4a385824"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:16:15.149314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.149198    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81be89f4-ebde-4d7b-8e24-18bd4a385824-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81be89f4-ebde-4d7b-8e24-18bd4a385824-istiod-ca-cert\") pod \"81be89f4-ebde-4d7b-8e24-18bd4a385824\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") "
Nov 01 04:16:15.149521 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.149385    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-certs\") pod \"81be89f4-ebde-4d7b-8e24-18bd4a385824\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") "
Nov 01 04:16:15.149521 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.149444    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-kube-api-access-gnsv6\" (UniqueName: \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-kube-api-access-gnsv6\") pod \"81be89f4-ebde-4d7b-8e24-18bd4a385824\" (UID: \"81be89f4-ebde-4d7b-8e24-18bd4a385824\") "
Nov 01 04:16:15.149521 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.149382    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-data" pod "81be89f4-ebde-4d7b-8e24-18bd4a385824" (UID: "81be89f4-ebde-4d7b-8e24-18bd4a385824"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:16:15.149521 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.149477    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-credential-socket" pod "81be89f4-ebde-4d7b-8e24-18bd4a385824" (UID: "81be89f4-ebde-4d7b-8e24-18bd4a385824"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:16:15.149734 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.149577    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81be89f4-ebde-4d7b-8e24-18bd4a385824-istiod-ca-cert" pod "81be89f4-ebde-4d7b-8e24-18bd4a385824" (UID: "81be89f4-ebde-4d7b-8e24-18bd4a385824"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:16:15.149734 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.149626    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-certs" pod "81be89f4-ebde-4d7b-8e24-18bd4a385824" (UID: "81be89f4-ebde-4d7b-8e24-18bd4a385824"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:16:15.150097 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.150071    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:15.150097 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.150096    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:15.150277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.150110    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81be89f4-ebde-4d7b-8e24-18bd4a385824-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:15.150277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.150127    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:15.150277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.150141    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:15.151291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.151264    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-podinfo" pod "81be89f4-ebde-4d7b-8e24-18bd4a385824" (UID: "81be89f4-ebde-4d7b-8e24-18bd4a385824"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:16:15.151376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.151314    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-envoy" pod "81be89f4-ebde-4d7b-8e24-18bd4a385824" (UID: "81be89f4-ebde-4d7b-8e24-18bd4a385824"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:16:15.151376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.151352    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-token" pod "81be89f4-ebde-4d7b-8e24-18bd4a385824" (UID: "81be89f4-ebde-4d7b-8e24-18bd4a385824"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:16:15.151609 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.151588    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-kube-api-access-gnsv6" pod "81be89f4-ebde-4d7b-8e24-18bd4a385824" (UID: "81be89f4-ebde-4d7b-8e24-18bd4a385824"). InnerVolumeSpecName "kube-api-access-gnsv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:16:15.251448 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.251400    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnsv6\" (UniqueName: \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-kube-api-access-gnsv6\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:15.251644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.251475    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:15.251644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.251491    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:15.251644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.251503    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81be89f4-ebde-4d7b-8e24-18bd4a385824-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:15.527491 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.527450    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"]
Nov 01 04:16:15.532358 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.532329    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-9tw2x"]
Nov 01 04:16:15.652974 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:15.652309    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" path="/var/lib/kubelet/pods/81be89f4-ebde-4d7b-8e24-18bd4a385824/volumes"
Nov 01 04:16:16.746004 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.745971    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"]
Nov 01 04:16:16.746538 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.746514    4882 cpu_manager.go:414] "containerMap: removing container" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy"
Nov 01 04:16:16.746538 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.746531    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy"
Nov 01 04:16:16.746730 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.746562    4882 cpu_manager.go:414] "containerMap: removing container" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy"
Nov 01 04:16:16.746730 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.746570    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy"
Nov 01 04:16:16.746730 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.746682    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy"
Nov 01 04:16:16.746730 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.746692    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="81be89f4-ebde-4d7b-8e24-18bd4a385824" containerName="istio-proxy"
Nov 01 04:16:16.751465 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.751444    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:16.846117 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.846082    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:16:16.846117 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.846085    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:16:16.851130 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.851107    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"]
Nov 01 04:16:16.870492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.870462    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2fgk\" (UniqueName: \"kubernetes.io/projected/b726ea1b-89fe-4045-b3bb-663943c12d2b-kube-api-access-j2fgk\") pod \"maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:16.870587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.870564    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:16.870699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.870679    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:16.972107 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.972081    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:16.972285 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.972160    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fgk\" (UniqueName: \"kubernetes.io/projected/b726ea1b-89fe-4045-b3bb-663943c12d2b-kube-api-access-j2fgk\") pod \"maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:16.972285 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.972257    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:16.972844 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.972816    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:16.974642 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:16.974622    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:17.028820 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:17.028760    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fgk\" (UniqueName: \"kubernetes.io/projected/b726ea1b-89fe-4045-b3bb-663943c12d2b-kube-api-access-j2fgk\") pod \"maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:17.065570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:17.065549    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:17.440032 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:17.440007    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"]
Nov 01 04:16:17.442403 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:16:17.442375    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb726ea1b_89fe_4045_b3bb_663943c12d2b.slice/crio-24f927371f53e726c932612a878227f30b9dd297d4e311b320e64b6de895e9a7 WatchSource:0}: Error finding container 24f927371f53e726c932612a878227f30b9dd297d4e311b320e64b6de895e9a7: Status 404 returned error can't find the container with id 24f927371f53e726c932612a878227f30b9dd297d4e311b320e64b6de895e9a7
Nov 01 04:16:18.129713 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:18.129618    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk" event={"ID":"b726ea1b-89fe-4045-b3bb-663943c12d2b","Type":"ContainerStarted","Data":"c6452093e1f44d7020502ae04d5b75722c12c77a2c98573a775c80e5c47cd80e"}
Nov 01 04:16:18.129713 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:18.129672    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk" event={"ID":"b726ea1b-89fe-4045-b3bb-663943c12d2b","Type":"ContainerStarted","Data":"24f927371f53e726c932612a878227f30b9dd297d4e311b320e64b6de895e9a7"}
Nov 01 04:16:18.238085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:18.238026    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk" podStartSLOduration=2.238012281 podStartE2EDuration="2.238012281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:16:18.237270293 +0000 UTC m=+2046.799303309" watchObservedRunningTime="2026-11-01 04:16:18.238012281 +0000 UTC m=+2046.800045299"
Nov 01 04:16:19.136769 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:19.136737    4882 generic.go:356] "Generic (PLEG): container finished" podID="b726ea1b-89fe-4045-b3bb-663943c12d2b" containerID="c6452093e1f44d7020502ae04d5b75722c12c77a2c98573a775c80e5c47cd80e" exitCode=0
Nov 01 04:16:19.137189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:19.136817    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk" event={"ID":"b726ea1b-89fe-4045-b3bb-663943c12d2b","Type":"ContainerDied","Data":"c6452093e1f44d7020502ae04d5b75722c12c77a2c98573a775c80e5c47cd80e"}
Nov 01 04:16:20.294185 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:20.294153    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:20.421908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:20.421823    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b726ea1b-89fe-4045-b3bb-663943c12d2b-kube-api-access-j2fgk\" (UniqueName: \"kubernetes.io/projected/b726ea1b-89fe-4045-b3bb-663943c12d2b-kube-api-access-j2fgk\") pod \"b726ea1b-89fe-4045-b3bb-663943c12d2b\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") "
Nov 01 04:16:20.422087 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:20.421914    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-xks-ca\") pod \"b726ea1b-89fe-4045-b3bb-663943c12d2b\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") "
Nov 01 04:16:20.422087 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:20.422045    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-openshift-ca\") pod \"b726ea1b-89fe-4045-b3bb-663943c12d2b\" (UID: \"b726ea1b-89fe-4045-b3bb-663943c12d2b\") "
Nov 01 04:16:20.422442 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:20.422399    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-openshift-ca" pod "b726ea1b-89fe-4045-b3bb-663943c12d2b" (UID: "b726ea1b-89fe-4045-b3bb-663943c12d2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:16:20.422542 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:20.422509    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:20.424132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:20.424100    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-xks-ca" pod "b726ea1b-89fe-4045-b3bb-663943c12d2b" (UID: "b726ea1b-89fe-4045-b3bb-663943c12d2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:16:20.424500 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:20.424477    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b726ea1b-89fe-4045-b3bb-663943c12d2b-kube-api-access-j2fgk" pod "b726ea1b-89fe-4045-b3bb-663943c12d2b" (UID: "b726ea1b-89fe-4045-b3bb-663943c12d2b"). InnerVolumeSpecName "kube-api-access-j2fgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:16:20.524047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:20.524013    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b726ea1b-89fe-4045-b3bb-663943c12d2b-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:20.524047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:20.524046    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2fgk\" (UniqueName: \"kubernetes.io/projected/b726ea1b-89fe-4045-b3bb-663943c12d2b-kube-api-access-j2fgk\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:21.155043 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:21.155015    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"
Nov 01 04:16:21.155236 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:21.155050    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk" event={"ID":"b726ea1b-89fe-4045-b3bb-663943c12d2b","Type":"ContainerDied","Data":"24f927371f53e726c932612a878227f30b9dd297d4e311b320e64b6de895e9a7"}
Nov 01 04:16:21.155236 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:21.155098    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24f927371f53e726c932612a878227f30b9dd297d4e311b320e64b6de895e9a7"
Nov 01 04:16:25.753066 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.753032    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"]
Nov 01 04:16:25.753960 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.753930    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b726ea1b-89fe-4045-b3bb-663943c12d2b" containerName="revoke-keys"
Nov 01 04:16:25.753960 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.753950    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b726ea1b-89fe-4045-b3bb-663943c12d2b" containerName="revoke-keys"
Nov 01 04:16:25.754107 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.754054    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b726ea1b-89fe-4045-b3bb-663943c12d2b" containerName="revoke-keys"
Nov 01 04:16:25.757704 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.757686    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:25.809167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.809137    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:16:25.809167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.809145    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:16:25.858715 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.858681    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"]
Nov 01 04:16:25.892411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.892381    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:25.892589 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.892458    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx9r9\" (UniqueName: \"kubernetes.io/projected/f4e1e789-8050-47f6-912f-a0e6239bb5d8-kube-api-access-kx9r9\") pod \"maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:25.892589 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.892538    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:25.993460 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.993397    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:25.993654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.993485    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx9r9\" (UniqueName: \"kubernetes.io/projected/f4e1e789-8050-47f6-912f-a0e6239bb5d8-kube-api-access-kx9r9\") pod \"maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:25.993654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.993524    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:25.994279 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.994257    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:25.995797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:25.995772    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:26.050766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:26.050693    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx9r9\" (UniqueName: \"kubernetes.io/projected/f4e1e789-8050-47f6-912f-a0e6239bb5d8-kube-api-access-kx9r9\") pod \"maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:26.071504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:26.071481    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:26.472031 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:26.472002    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"]
Nov 01 04:16:26.473050 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:16:26.473025    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e1e789_8050_47f6_912f_a0e6239bb5d8.slice/crio-2375f5fdf3f10a306587feb53e17a5db5f89fb0b9ffb0d86553a4366165f91ed WatchSource:0}: Error finding container 2375f5fdf3f10a306587feb53e17a5db5f89fb0b9ffb0d86553a4366165f91ed: Status 404 returned error can't find the container with id 2375f5fdf3f10a306587feb53e17a5db5f89fb0b9ffb0d86553a4366165f91ed
Nov 01 04:16:27.196142 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:27.196108    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6" event={"ID":"f4e1e789-8050-47f6-912f-a0e6239bb5d8","Type":"ContainerStarted","Data":"87ae0c7f5a00f62895b0c7a8c1fcf7c1bc4b760d27069c8199f04c8bcb846d31"}
Nov 01 04:16:27.196565 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:27.196156    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6" event={"ID":"f4e1e789-8050-47f6-912f-a0e6239bb5d8","Type":"ContainerStarted","Data":"2375f5fdf3f10a306587feb53e17a5db5f89fb0b9ffb0d86553a4366165f91ed"}
Nov 01 04:16:27.307180 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:27.307122    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6" podStartSLOduration=2.307105509 podStartE2EDuration="2.307105509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:16:27.30374917 +0000 UTC m=+2055.865782186" watchObservedRunningTime="2026-11-01 04:16:27.307105509 +0000 UTC m=+2055.869138525"
Nov 01 04:16:28.216046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:28.216011    4882 generic.go:356] "Generic (PLEG): container finished" podID="f4e1e789-8050-47f6-912f-a0e6239bb5d8" containerID="87ae0c7f5a00f62895b0c7a8c1fcf7c1bc4b760d27069c8199f04c8bcb846d31" exitCode=0
Nov 01 04:16:28.216504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:28.216100    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6" event={"ID":"f4e1e789-8050-47f6-912f-a0e6239bb5d8","Type":"ContainerDied","Data":"87ae0c7f5a00f62895b0c7a8c1fcf7c1bc4b760d27069c8199f04c8bcb846d31"}
Nov 01 04:16:28.724219 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:16:28.724183    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:16:29.368605 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:29.368581    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:29.552945 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:29.552865    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e1e789-8050-47f6-912f-a0e6239bb5d8-kube-api-access-kx9r9\" (UniqueName: \"kubernetes.io/projected/f4e1e789-8050-47f6-912f-a0e6239bb5d8-kube-api-access-kx9r9\") pod \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") "
Nov 01 04:16:29.553100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:29.552954    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-xks-ca\") pod \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") "
Nov 01 04:16:29.553100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:29.553061    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-openshift-ca\") pod \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\" (UID: \"f4e1e789-8050-47f6-912f-a0e6239bb5d8\") "
Nov 01 04:16:29.553406 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:29.553381    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-openshift-ca" pod "f4e1e789-8050-47f6-912f-a0e6239bb5d8" (UID: "f4e1e789-8050-47f6-912f-a0e6239bb5d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:16:29.553608 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:29.553589    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:29.555067 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:29.555038    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-xks-ca" pod "f4e1e789-8050-47f6-912f-a0e6239bb5d8" (UID: "f4e1e789-8050-47f6-912f-a0e6239bb5d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:16:29.555158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:29.555092    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e1e789-8050-47f6-912f-a0e6239bb5d8-kube-api-access-kx9r9" pod "f4e1e789-8050-47f6-912f-a0e6239bb5d8" (UID: "f4e1e789-8050-47f6-912f-a0e6239bb5d8"). InnerVolumeSpecName "kube-api-access-kx9r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:16:29.655214 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:29.655184    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx9r9\" (UniqueName: \"kubernetes.io/projected/f4e1e789-8050-47f6-912f-a0e6239bb5d8-kube-api-access-kx9r9\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:29.655214 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:29.655212    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e1e789-8050-47f6-912f-a0e6239bb5d8-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:30.230388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:30.230355    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6" event={"ID":"f4e1e789-8050-47f6-912f-a0e6239bb5d8","Type":"ContainerDied","Data":"2375f5fdf3f10a306587feb53e17a5db5f89fb0b9ffb0d86553a4366165f91ed"}
Nov 01 04:16:30.230552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:30.230399    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"
Nov 01 04:16:30.230552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:30.230403    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2375f5fdf3f10a306587feb53e17a5db5f89fb0b9ffb0d86553a4366165f91ed"
Nov 01 04:16:42.170644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:42.170602    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"]
Nov 01 04:16:42.216593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:42.216550    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-157fed-9bjb6"]
Nov 01 04:16:43.601003 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:43.600967    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e1e789-8050-47f6-912f-a0e6239bb5d8" path="/var/lib/kubelet/pods/f4e1e789-8050-47f6-912f-a0e6239bb5d8/volumes"
Nov 01 04:16:46.922806 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:46.922765    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"]
Nov 01 04:16:46.923313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:46.923058    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr" podUID="1c49bad6-a6a2-4766-8d4c-309d77e13eff" containerName="payload-processing" containerID="cri-o://40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04" gracePeriod=30
Nov 01 04:16:47.028367 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.028332    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"]
Nov 01 04:16:47.028631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.028582    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z" podUID="8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa" containerName="payload-pre-processing" containerID="cri-o://3c8842398b7d6d0f7b634c3c14c06f2c27f9f6e907193c56003c9a31ac7aacaf" gracePeriod=30
Nov 01 04:16:47.304835 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.304788    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:16:47.341401 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.341356    4882 generic.go:356] "Generic (PLEG): container finished" podID="8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa" containerID="3c8842398b7d6d0f7b634c3c14c06f2c27f9f6e907193c56003c9a31ac7aacaf" exitCode=0
Nov 01 04:16:47.341785 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.341524    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z" event={"ID":"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa","Type":"ContainerDied","Data":"3c8842398b7d6d0f7b634c3c14c06f2c27f9f6e907193c56003c9a31ac7aacaf"}
Nov 01 04:16:47.343476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.343406    4882 generic.go:356] "Generic (PLEG): container finished" podID="1c49bad6-a6a2-4766-8d4c-309d77e13eff" containerID="40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04" exitCode=0
Nov 01 04:16:47.343594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.343482    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"
Nov 01 04:16:47.343594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.343506    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr" event={"ID":"1c49bad6-a6a2-4766-8d4c-309d77e13eff","Type":"ContainerDied","Data":"40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04"}
Nov 01 04:16:47.343594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.343568    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr" event={"ID":"1c49bad6-a6a2-4766-8d4c-309d77e13eff","Type":"ContainerDied","Data":"46f5089147c6d35f07afb5d0f608382fc8a6911f5027e014869f0c1bb90b534c"}
Nov 01 04:16:47.343731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.343597    4882 scope.go:122] "RemoveContainer" containerID="40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04"
Nov 01 04:16:47.354918 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.354897    4882 scope.go:122] "RemoveContainer" containerID="40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04"
Nov 01 04:16:47.355311 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:16:47.355273    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04\": container with ID starting with 40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04 not found: ID does not exist" containerID="40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04"
Nov 01 04:16:47.355476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.355313    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04"} err="failed to get container status \"40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04\": rpc error: code = NotFound desc = could not find container \"40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04\": container with ID starting with 40a1a073c0370987d73efd5d45695f89330ec9a0a0005990ab4c2f344cb05e04 not found: ID does not exist"
Nov 01 04:16:47.390561 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.387628    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:16:47.419013 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.418966    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c49bad6-a6a2-4766-8d4c-309d77e13eff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c49bad6-a6a2-4766-8d4c-309d77e13eff-plugins-config-volume\") pod \"1c49bad6-a6a2-4766-8d4c-309d77e13eff\" (UID: \"1c49bad6-a6a2-4766-8d4c-309d77e13eff\") "
Nov 01 04:16:47.419502 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.419470    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c49bad6-a6a2-4766-8d4c-309d77e13eff-kube-api-access-cpn97\" (UniqueName: \"kubernetes.io/projected/1c49bad6-a6a2-4766-8d4c-309d77e13eff-kube-api-access-cpn97\") pod \"1c49bad6-a6a2-4766-8d4c-309d77e13eff\" (UID: \"1c49bad6-a6a2-4766-8d4c-309d77e13eff\") "
Nov 01 04:16:47.420179 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.420145    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c49bad6-a6a2-4766-8d4c-309d77e13eff-plugins-config-volume" pod "1c49bad6-a6a2-4766-8d4c-309d77e13eff" (UID: "1c49bad6-a6a2-4766-8d4c-309d77e13eff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:16:47.420756 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.420730    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c49bad6-a6a2-4766-8d4c-309d77e13eff-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:47.422411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.422378    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c49bad6-a6a2-4766-8d4c-309d77e13eff-kube-api-access-cpn97" pod "1c49bad6-a6a2-4766-8d4c-309d77e13eff" (UID: "1c49bad6-a6a2-4766-8d4c-309d77e13eff"). InnerVolumeSpecName "kube-api-access-cpn97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:16:47.532554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.528011    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-kube-api-access-qbl46\" (UniqueName: \"kubernetes.io/projected/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-kube-api-access-qbl46\") pod \"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa\" (UID: \"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa\") "
Nov 01 04:16:47.532554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.528169    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-plugins-config-volume\") pod \"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa\" (UID: \"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa\") "
Nov 01 04:16:47.554473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.551909    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpn97\" (UniqueName: \"kubernetes.io/projected/1c49bad6-a6a2-4766-8d4c-309d77e13eff-kube-api-access-cpn97\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:47.554664 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.554611    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-kube-api-access-qbl46" pod "8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa" (UID: "8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa"). InnerVolumeSpecName "kube-api-access-qbl46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:16:47.575012 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.574655    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-plugins-config-volume" pod "8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa" (UID: "8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:16:47.653092 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.653056    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:47.653092 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.653091    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbl46\" (UniqueName: \"kubernetes.io/projected/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa-kube-api-access-qbl46\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:47.754146 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.754101    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"]
Nov 01 04:16:47.760836 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:47.760801    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-6f5b6b687d-lxzrr"]
Nov 01 04:16:48.386499 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:48.378311    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z" event={"ID":"8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa","Type":"ContainerDied","Data":"ab32fc65187fd609ff38380d7294ac555dfd5239b0c379214ec29a5d4b95796f"}
Nov 01 04:16:48.386499 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:48.378384    4882 scope.go:122] "RemoveContainer" containerID="3c8842398b7d6d0f7b634c3c14c06f2c27f9f6e907193c56003c9a31ac7aacaf"
Nov 01 04:16:48.386499 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:48.378975    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"
Nov 01 04:16:48.533738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:48.533705    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"]
Nov 01 04:16:48.537791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:48.537764    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-cd8d5c94c-2fg4z"]
Nov 01 04:16:48.695681 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:48.695636    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"]
Nov 01 04:16:48.698969 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:48.698938    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5b71cd-trcdk"]
Nov 01 04:16:49.410121 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:49.410088    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"]
Nov 01 04:16:49.410607 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:49.410294    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy" containerID="cri-o://b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053" gracePeriod=30
Nov 01 04:16:49.601764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:49.601049    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c49bad6-a6a2-4766-8d4c-309d77e13eff" path="/var/lib/kubelet/pods/1c49bad6-a6a2-4766-8d4c-309d77e13eff/volumes"
Nov 01 04:16:49.601764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:49.601471    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa" path="/var/lib/kubelet/pods/8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa/volumes"
Nov 01 04:16:49.602148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:49.601863    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b726ea1b-89fe-4045-b3bb-663943c12d2b" path="/var/lib/kubelet/pods/b726ea1b-89fe-4045-b3bb-663943c12d2b/volumes"
Nov 01 04:16:50.298582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.298550    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"]
Nov 01 04:16:50.299074 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.299047    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa" containerName="payload-pre-processing"
Nov 01 04:16:50.299074 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.299063    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa" containerName="payload-pre-processing"
Nov 01 04:16:50.299287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.299091    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f4e1e789-8050-47f6-912f-a0e6239bb5d8" containerName="revoke-keys"
Nov 01 04:16:50.299287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.299099    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e1e789-8050-47f6-912f-a0e6239bb5d8" containerName="revoke-keys"
Nov 01 04:16:50.299287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.299129    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1c49bad6-a6a2-4766-8d4c-309d77e13eff" containerName="payload-processing"
Nov 01 04:16:50.299287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.299147    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c49bad6-a6a2-4766-8d4c-309d77e13eff" containerName="payload-processing"
Nov 01 04:16:50.299502 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.299309    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b64c7a6-19a3-4f3d-b771-4e3fb92af7fa" containerName="payload-pre-processing"
Nov 01 04:16:50.299502 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.299328    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e1e789-8050-47f6-912f-a0e6239bb5d8" containerName="revoke-keys"
Nov 01 04:16:50.299502 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.299342    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c49bad6-a6a2-4766-8d4c-309d77e13eff" containerName="payload-processing"
Nov 01 04:16:50.304424 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.304401    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:50.357715 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.357679    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"]
Nov 01 04:16:50.397979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.397945    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7802f48-af03-4377-929e-7685c26a964c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:50.398160 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.397989    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7802f48-af03-4377-929e-7685c26a964c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:50.398228 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.398171    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt7ms\" (UniqueName: \"kubernetes.io/projected/c7802f48-af03-4377-929e-7685c26a964c-kube-api-access-lt7ms\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:50.398912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.398893    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:16:50.398976 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.398936    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:16:50.499457 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.499407    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7802f48-af03-4377-929e-7685c26a964c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:50.499963 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.499480    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7802f48-af03-4377-929e-7685c26a964c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:50.499963 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.499576    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt7ms\" (UniqueName: \"kubernetes.io/projected/c7802f48-af03-4377-929e-7685c26a964c-kube-api-access-lt7ms\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:50.500086 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.500061    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7802f48-af03-4377-929e-7685c26a964c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:50.502029 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.502008    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7802f48-af03-4377-929e-7685c26a964c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:50.557023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.556949    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt7ms\" (UniqueName: \"kubernetes.io/projected/c7802f48-af03-4377-929e-7685c26a964c-kube-api-access-lt7ms\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:50.617340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.617309    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:50.986585 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:50.986547    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"]
Nov 01 04:16:50.987968 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:16:50.987936    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7802f48_af03_4377_929e_7685c26a964c.slice/crio-3c07fc628f3ec8535ad09d1f3b30e6d60d0dd4fdb388f9260faada74212a157f WatchSource:0}: Error finding container 3c07fc628f3ec8535ad09d1f3b30e6d60d0dd4fdb388f9260faada74212a157f: Status 404 returned error can't find the container with id 3c07fc628f3ec8535ad09d1f3b30e6d60d0dd4fdb388f9260faada74212a157f
Nov 01 04:16:51.410395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:51.410359    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5" event={"ID":"c7802f48-af03-4377-929e-7685c26a964c","Type":"ContainerStarted","Data":"e379027c80a1f90e23008a89f71965a2b25c58376e74cd704fac3711039b477c"}
Nov 01 04:16:51.410590 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:51.410415    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5" event={"ID":"c7802f48-af03-4377-929e-7685c26a964c","Type":"ContainerStarted","Data":"3c07fc628f3ec8535ad09d1f3b30e6d60d0dd4fdb388f9260faada74212a157f"}
Nov 01 04:16:51.533503 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:51.530380    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5" podStartSLOduration=1.5303617 podStartE2EDuration="1.5303617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:16:51.519847936 +0000 UTC m=+2080.081880972" watchObservedRunningTime="2026-11-01 04:16:51.5303617 +0000 UTC m=+2080.092394718"
Nov 01 04:16:52.417967 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:52.417929    4882 generic.go:356] "Generic (PLEG): container finished" podID="c7802f48-af03-4377-929e-7685c26a964c" containerID="e379027c80a1f90e23008a89f71965a2b25c58376e74cd704fac3711039b477c" exitCode=0
Nov 01 04:16:52.418146 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:52.418009    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5" event={"ID":"c7802f48-af03-4377-929e-7685c26a964c","Type":"ContainerDied","Data":"e379027c80a1f90e23008a89f71965a2b25c58376e74cd704fac3711039b477c"}
Nov 01 04:16:53.072053 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.072015    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:16:53.072459 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.072079    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:16:53.581091 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.581062    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:53.653917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.653891    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7802f48-af03-4377-929e-7685c26a964c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7802f48-af03-4377-929e-7685c26a964c-maas-api-openshift-ca\") pod \"c7802f48-af03-4377-929e-7685c26a964c\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") "
Nov 01 04:16:53.654072 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.653940    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7802f48-af03-4377-929e-7685c26a964c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7802f48-af03-4377-929e-7685c26a964c-maas-api-xks-ca\") pod \"c7802f48-af03-4377-929e-7685c26a964c\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") "
Nov 01 04:16:53.654072 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.653977    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7802f48-af03-4377-929e-7685c26a964c-kube-api-access-lt7ms\" (UniqueName: \"kubernetes.io/projected/c7802f48-af03-4377-929e-7685c26a964c-kube-api-access-lt7ms\") pod \"c7802f48-af03-4377-929e-7685c26a964c\" (UID: \"c7802f48-af03-4377-929e-7685c26a964c\") "
Nov 01 04:16:53.654296 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.654266    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7802f48-af03-4377-929e-7685c26a964c-maas-api-openshift-ca" pod "c7802f48-af03-4377-929e-7685c26a964c" (UID: "c7802f48-af03-4377-929e-7685c26a964c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:16:53.656159 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.656129    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7802f48-af03-4377-929e-7685c26a964c-maas-api-xks-ca" pod "c7802f48-af03-4377-929e-7685c26a964c" (UID: "c7802f48-af03-4377-929e-7685c26a964c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:16:53.656239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.656200    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7802f48-af03-4377-929e-7685c26a964c-kube-api-access-lt7ms" pod "c7802f48-af03-4377-929e-7685c26a964c" (UID: "c7802f48-af03-4377-929e-7685c26a964c"). InnerVolumeSpecName "kube-api-access-lt7ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:16:53.755396 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.755359    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7802f48-af03-4377-929e-7685c26a964c-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:53.755396 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.755387    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7802f48-af03-4377-929e-7685c26a964c-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:53.755396 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:53.755399    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt7ms\" (UniqueName: \"kubernetes.io/projected/c7802f48-af03-4377-929e-7685c26a964c-kube-api-access-lt7ms\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:54.432353 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.432327    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"
Nov 01 04:16:54.432739 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.432366    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5" event={"ID":"c7802f48-af03-4377-929e-7685c26a964c","Type":"ContainerDied","Data":"3c07fc628f3ec8535ad09d1f3b30e6d60d0dd4fdb388f9260faada74212a157f"}
Nov 01 04:16:54.432739 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.432421    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c07fc628f3ec8535ad09d1f3b30e6d60d0dd4fdb388f9260faada74212a157f"
Nov 01 04:16:54.672750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.672693    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:16:54.768110 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768076    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-token\" (UniqueName: \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-token\") pod \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") "
Nov 01 04:16:54.768277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768174    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-podinfo\") pod \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") "
Nov 01 04:16:54.768277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768228    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-kube-api-access-t2pwc\" (UniqueName: \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-kube-api-access-t2pwc\") pod \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") "
Nov 01 04:16:54.768359 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768308    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-credential-socket\") pod \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") "
Nov 01 04:16:54.768413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768354    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-certs\") pod \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") "
Nov 01 04:16:54.768413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768389    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-socket\") pod \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") "
Nov 01 04:16:54.768537 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768461    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-envoy\") pod \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") "
Nov 01 04:16:54.768537 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768501    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istiod-ca-cert\") pod \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") "
Nov 01 04:16:54.768643 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768567    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-data\") pod \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\" (UID: \"865370fa-7d36-46e3-98e2-88ca8d35c6b5\") "
Nov 01 04:16:54.768698 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768643    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-socket" pod "865370fa-7d36-46e3-98e2-88ca8d35c6b5" (UID: "865370fa-7d36-46e3-98e2-88ca8d35c6b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:16:54.768757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768686    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-certs" pod "865370fa-7d36-46e3-98e2-88ca8d35c6b5" (UID: "865370fa-7d36-46e3-98e2-88ca8d35c6b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:16:54.768757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768695    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-credential-socket" pod "865370fa-7d36-46e3-98e2-88ca8d35c6b5" (UID: "865370fa-7d36-46e3-98e2-88ca8d35c6b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:16:54.768994 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768972    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-data" pod "865370fa-7d36-46e3-98e2-88ca8d35c6b5" (UID: "865370fa-7d36-46e3-98e2-88ca8d35c6b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:16:54.769093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.768988    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istiod-ca-cert" pod "865370fa-7d36-46e3-98e2-88ca8d35c6b5" (UID: "865370fa-7d36-46e3-98e2-88ca8d35c6b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:16:54.769178 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.769153    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:54.769313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.769181    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:54.769313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.769196    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:54.769313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.769210    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:54.769313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.769224    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:54.770464 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.770439    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-token" pod "865370fa-7d36-46e3-98e2-88ca8d35c6b5" (UID: "865370fa-7d36-46e3-98e2-88ca8d35c6b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:16:54.770757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.770737    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-kube-api-access-t2pwc" pod "865370fa-7d36-46e3-98e2-88ca8d35c6b5" (UID: "865370fa-7d36-46e3-98e2-88ca8d35c6b5"). InnerVolumeSpecName "kube-api-access-t2pwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:16:54.770882 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.770866    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-podinfo" pod "865370fa-7d36-46e3-98e2-88ca8d35c6b5" (UID: "865370fa-7d36-46e3-98e2-88ca8d35c6b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:16:54.770969 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.770956    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-envoy" pod "865370fa-7d36-46e3-98e2-88ca8d35c6b5" (UID: "865370fa-7d36-46e3-98e2-88ca8d35c6b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:16:54.870852 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.870820    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:54.870852 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.870849    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2pwc\" (UniqueName: \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-kube-api-access-t2pwc\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:54.870852 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.870860    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:54.871081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:54.870870    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865370fa-7d36-46e3-98e2-88ca8d35c6b5-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:55.440657 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.440623    4882 generic.go:356] "Generic (PLEG): container finished" podID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerID="b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053" exitCode=0
Nov 01 04:16:55.441100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.440679    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"
Nov 01 04:16:55.441100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.440703    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" event={"ID":"865370fa-7d36-46e3-98e2-88ca8d35c6b5","Type":"ContainerDied","Data":"b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053"}
Nov 01 04:16:55.441100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.440757    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n" event={"ID":"865370fa-7d36-46e3-98e2-88ca8d35c6b5","Type":"ContainerDied","Data":"137a71980e4a93b0bd8a8a60d6ee1c3d683dbbdd6c17fa116dbb3a2c30514fb2"}
Nov 01 04:16:55.441100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.440781    4882 scope.go:122] "RemoveContainer" containerID="b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053"
Nov 01 04:16:55.451360 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.451342    4882 scope.go:122] "RemoveContainer" containerID="cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01"
Nov 01 04:16:55.460254 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.460238    4882 scope.go:122] "RemoveContainer" containerID="b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053"
Nov 01 04:16:55.460526 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:16:55.460507    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053\": container with ID starting with b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053 not found: ID does not exist" containerID="b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053"
Nov 01 04:16:55.460592 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.460534    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053"} err="failed to get container status \"b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053\": rpc error: code = NotFound desc = could not find container \"b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053\": container with ID starting with b7b77e17b9f2a9fc9974ebde9be3191956741880c6c9e61e819120b681cea053 not found: ID does not exist"
Nov 01 04:16:55.460592 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.460549    4882 scope.go:122] "RemoveContainer" containerID="cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01"
Nov 01 04:16:55.460769 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:16:55.460749    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01\": container with ID starting with cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01 not found: ID does not exist" containerID="cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01"
Nov 01 04:16:55.460827 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.460776    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01"} err="failed to get container status \"cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01\": rpc error: code = NotFound desc = could not find container \"cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01\": container with ID starting with cc71b2d387626693cdcbf2aa5b3f2bc875fd405fe665e2ae17e2b2f40a088d01 not found: ID does not exist"
Nov 01 04:16:55.607124 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.607085    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"]
Nov 01 04:16:55.653862 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:55.653825    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-157fed-openshift-default-5975bb79d7-d5t8n"]
Nov 01 04:16:56.650753 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.650719    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"]
Nov 01 04:16:56.651252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.651239    4882 cpu_manager.go:414] "containerMap: removing container" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy"
Nov 01 04:16:56.651314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.651254    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy"
Nov 01 04:16:56.651314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.651265    4882 cpu_manager.go:414] "containerMap: removing container" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy"
Nov 01 04:16:56.651314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.651271    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy"
Nov 01 04:16:56.651417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.651320    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c7802f48-af03-4377-929e-7685c26a964c" containerName="revoke-keys"
Nov 01 04:16:56.651417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.651326    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7802f48-af03-4377-929e-7685c26a964c" containerName="revoke-keys"
Nov 01 04:16:56.651417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.651407    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7802f48-af03-4377-929e-7685c26a964c" containerName="revoke-keys"
Nov 01 04:16:56.651417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.651416    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy"
Nov 01 04:16:56.651572 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.651423    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" containerName="istio-proxy"
Nov 01 04:16:56.656276 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.656256    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:56.705192 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.705164    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:16:56.751302 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.751254    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:16:56.754554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.754525    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"]
Nov 01 04:16:56.799897 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.799869    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrw9g\" (UniqueName: \"kubernetes.io/projected/efd3dc33-6eb3-424f-8b25-678e7ea636c9-kube-api-access-mrw9g\") pod \"maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:56.800044 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.799938    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:56.800092 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.800072    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:56.900856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.900770    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:56.901032 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.900888    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrw9g\" (UniqueName: \"kubernetes.io/projected/efd3dc33-6eb3-424f-8b25-678e7ea636c9-kube-api-access-mrw9g\") pod \"maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:56.901032 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.900969    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:56.901395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.901374    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:56.903256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.903237    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:56.957956 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.957924    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrw9g\" (UniqueName: \"kubernetes.io/projected/efd3dc33-6eb3-424f-8b25-678e7ea636c9-kube-api-access-mrw9g\") pod \"maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:56.969277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:56.969255    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:57.339540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:57.339511    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"]
Nov 01 04:16:57.341464 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:16:57.341405    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd3dc33_6eb3_424f_8b25_678e7ea636c9.slice/crio-f1b31c992f6b345eba9b0f406e67fd775fd2ba02995c5eb3e1e01b3950561415 WatchSource:0}: Error finding container f1b31c992f6b345eba9b0f406e67fd775fd2ba02995c5eb3e1e01b3950561415: Status 404 returned error can't find the container with id f1b31c992f6b345eba9b0f406e67fd775fd2ba02995c5eb3e1e01b3950561415
Nov 01 04:16:57.456869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:57.456837    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj" event={"ID":"efd3dc33-6eb3-424f-8b25-678e7ea636c9","Type":"ContainerStarted","Data":"f1b31c992f6b345eba9b0f406e67fd775fd2ba02995c5eb3e1e01b3950561415"}
Nov 01 04:16:57.600527 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:57.600453    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865370fa-7d36-46e3-98e2-88ca8d35c6b5" path="/var/lib/kubelet/pods/865370fa-7d36-46e3-98e2-88ca8d35c6b5/volumes"
Nov 01 04:16:58.465602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:58.465565    4882 generic.go:356] "Generic (PLEG): container finished" podID="efd3dc33-6eb3-424f-8b25-678e7ea636c9" containerID="a33071fec723ca0c7fe1b1ae6b9f77892281915bc8d19da81296f16100d3264e" exitCode=0
Nov 01 04:16:58.466004 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:58.465636    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj" event={"ID":"efd3dc33-6eb3-424f-8b25-678e7ea636c9","Type":"ContainerDied","Data":"a33071fec723ca0c7fe1b1ae6b9f77892281915bc8d19da81296f16100d3264e"}
Nov 01 04:16:58.725057 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:16:58.725014    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:16:59.632699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:59.632672    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:16:59.751979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:59.751894    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-xks-ca\") pod \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") "
Nov 01 04:16:59.752194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:59.752173    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-openshift-ca\") pod \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") "
Nov 01 04:16:59.752261 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:59.752228    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd3dc33-6eb3-424f-8b25-678e7ea636c9-kube-api-access-mrw9g\" (UniqueName: \"kubernetes.io/projected/efd3dc33-6eb3-424f-8b25-678e7ea636c9-kube-api-access-mrw9g\") pod \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\" (UID: \"efd3dc33-6eb3-424f-8b25-678e7ea636c9\") "
Nov 01 04:16:59.752679 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:59.752653    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-openshift-ca" pod "efd3dc33-6eb3-424f-8b25-678e7ea636c9" (UID: "efd3dc33-6eb3-424f-8b25-678e7ea636c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:16:59.753130 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:59.753106    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:59.755196 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:59.755168    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd3dc33-6eb3-424f-8b25-678e7ea636c9-kube-api-access-mrw9g" pod "efd3dc33-6eb3-424f-8b25-678e7ea636c9" (UID: "efd3dc33-6eb3-424f-8b25-678e7ea636c9"). InnerVolumeSpecName "kube-api-access-mrw9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:16:59.755791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:59.755762    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-xks-ca" pod "efd3dc33-6eb3-424f-8b25-678e7ea636c9" (UID: "efd3dc33-6eb3-424f-8b25-678e7ea636c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:16:59.854336 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:59.854300    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrw9g\" (UniqueName: \"kubernetes.io/projected/efd3dc33-6eb3-424f-8b25-678e7ea636c9-kube-api-access-mrw9g\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:16:59.854336 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:16:59.854331    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd3dc33-6eb3-424f-8b25-678e7ea636c9-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:17:00.479668 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:00.479635    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"
Nov 01 04:17:00.479855 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:00.479670    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj" event={"ID":"efd3dc33-6eb3-424f-8b25-678e7ea636c9","Type":"ContainerDied","Data":"f1b31c992f6b345eba9b0f406e67fd775fd2ba02995c5eb3e1e01b3950561415"}
Nov 01 04:17:00.479855 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:00.479718    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1b31c992f6b345eba9b0f406e67fd775fd2ba02995c5eb3e1e01b3950561415"
Nov 01 04:17:09.586943 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:09.586906    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"]
Nov 01 04:17:09.587401 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:09.587173    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerName="istio-proxy" containerID="cri-o://a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9" gracePeriod=30
Nov 01 04:17:11.500663 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:11.500611    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:17:11.501385 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:11.501345    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:17:12.986785 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:12.986751    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:17:12.989249 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:12.989225    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:17:13.215418 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:13.215385    4882 scope.go:122] "RemoveContainer" containerID="a39bfdcf71bfd7ffe1d3bf9f18fdf1a8c228a8f0353d7f2ca70ac9a5319be084"
Nov 01 04:17:13.316227 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:13.316185    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"]
Nov 01 04:17:13.359149 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:13.359109    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-535e84-xcmmj"]
Nov 01 04:17:13.600022 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:13.599934    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd3dc33-6eb3-424f-8b25-678e7ea636c9" path="/var/lib/kubelet/pods/efd3dc33-6eb3-424f-8b25-678e7ea636c9/volumes"
Nov 01 04:17:14.875296 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.875227    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:17:14.968971 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.968938    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-podinfo\") pod \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") "
Nov 01 04:17:14.969154 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.968988    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-socket\") pod \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") "
Nov 01 04:17:14.969198 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.969178    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-envoy\") pod \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") "
Nov 01 04:17:14.969274 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.969254    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-socket" pod "f715c6b5-edcb-46d9-8ed0-12c09b5164e7" (UID: "f715c6b5-edcb-46d9-8ed0-12c09b5164e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:17:14.969318 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.969261    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istiod-ca-cert\") pod \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") "
Nov 01 04:17:14.969440 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.969404    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-kube-api-access-g9gqj\" (UniqueName: \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-kube-api-access-g9gqj\") pod \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") "
Nov 01 04:17:14.969572 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.969549    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-data\") pod \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") "
Nov 01 04:17:14.969724 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.969607    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istiod-ca-cert" pod "f715c6b5-edcb-46d9-8ed0-12c09b5164e7" (UID: "f715c6b5-edcb-46d9-8ed0-12c09b5164e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:17:14.969724 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.969629    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-certs\") pod \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") "
Nov 01 04:17:14.969851 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.969779    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-credential-socket\") pod \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") "
Nov 01 04:17:14.969851 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.969827    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-token\" (UniqueName: \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-token\") pod \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\" (UID: \"f715c6b5-edcb-46d9-8ed0-12c09b5164e7\") "
Nov 01 04:17:14.969979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.969885    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-data" pod "f715c6b5-edcb-46d9-8ed0-12c09b5164e7" (UID: "f715c6b5-edcb-46d9-8ed0-12c09b5164e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:17:14.969979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.969905    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-certs" pod "f715c6b5-edcb-46d9-8ed0-12c09b5164e7" (UID: "f715c6b5-edcb-46d9-8ed0-12c09b5164e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:17:14.970125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.970093    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-credential-socket" pod "f715c6b5-edcb-46d9-8ed0-12c09b5164e7" (UID: "f715c6b5-edcb-46d9-8ed0-12c09b5164e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:17:14.970387 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.970368    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:17:14.970493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.970393    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:17:14.970493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.970407    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:17:14.970493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.970421    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:17:14.970493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.970483    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:17:14.971577 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.971547    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-envoy" pod "f715c6b5-edcb-46d9-8ed0-12c09b5164e7" (UID: "f715c6b5-edcb-46d9-8ed0-12c09b5164e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:17:14.971791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.971770    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-podinfo" pod "f715c6b5-edcb-46d9-8ed0-12c09b5164e7" (UID: "f715c6b5-edcb-46d9-8ed0-12c09b5164e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:17:14.971943 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.971921    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-kube-api-access-g9gqj" pod "f715c6b5-edcb-46d9-8ed0-12c09b5164e7" (UID: "f715c6b5-edcb-46d9-8ed0-12c09b5164e7"). InnerVolumeSpecName "kube-api-access-g9gqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:17:14.972042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:14.972025    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-token" pod "f715c6b5-edcb-46d9-8ed0-12c09b5164e7" (UID: "f715c6b5-edcb-46d9-8ed0-12c09b5164e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:17:15.074315 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.074282    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:17:15.074315 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.074313    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9gqj\" (UniqueName: \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-kube-api-access-g9gqj\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:17:15.074548 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.074324    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:17:15.074548 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.074334    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f715c6b5-edcb-46d9-8ed0-12c09b5164e7-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:17:15.575292 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.575248    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"]
Nov 01 04:17:15.575997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.575974    4882 cpu_manager.go:414] "containerMap: removing container" podUID="efd3dc33-6eb3-424f-8b25-678e7ea636c9" containerName="revoke-keys"
Nov 01 04:17:15.576169 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.576142    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd3dc33-6eb3-424f-8b25-678e7ea636c9" containerName="revoke-keys"
Nov 01 04:17:15.576291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.576183    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerName="istio-proxy"
Nov 01 04:17:15.576291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.576191    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerName="istio-proxy"
Nov 01 04:17:15.576395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.576346    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerName="istio-proxy"
Nov 01 04:17:15.576395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.576376    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd3dc33-6eb3-424f-8b25-678e7ea636c9" containerName="revoke-keys"
Nov 01 04:17:15.580184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.580157    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.597416 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.597374    4882 generic.go:356] "Generic (PLEG): container finished" podID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" containerID="a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9" exitCode=0
Nov 01 04:17:15.597599 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.597526    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"
Nov 01 04:17:15.604571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.604542    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" event={"ID":"f715c6b5-edcb-46d9-8ed0-12c09b5164e7","Type":"ContainerDied","Data":"a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9"}
Nov 01 04:17:15.604737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.604590    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh" event={"ID":"f715c6b5-edcb-46d9-8ed0-12c09b5164e7","Type":"ContainerDied","Data":"50730afc25f74f74d0f42208d95e8a18dfb3a9c2d9fa1167bfe3813c6514b965"}
Nov 01 04:17:15.604737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.604623    4882 scope.go:122] "RemoveContainer" containerID="a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9"
Nov 01 04:17:15.621146 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.621116    4882 scope.go:122] "RemoveContainer" containerID="a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9"
Nov 01 04:17:15.621592 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:17:15.621540    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9\": container with ID starting with a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9 not found: ID does not exist" containerID="a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9"
Nov 01 04:17:15.621711 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.621586    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9"} err="failed to get container status \"a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9\": rpc error: code = NotFound desc = could not find container \"a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9\": container with ID starting with a85867fc0129d171157b0f9561590b86c4a043b761f889c0f1e0e257d09344b9 not found: ID does not exist"
Nov 01 04:17:15.628918 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.628892    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a7b5f0-openshift-default-dockercfg-nstj2\""
Nov 01 04:17:15.648470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.647780    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"]
Nov 01 04:17:15.682179 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.682142    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7865f262-e4a2-4315-89c3-8e5655f1e47f-istiod-ca-cert\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.682359 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.682184    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-data\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.682359 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.682290    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-token\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.682508 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.682485    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cmv6\" (UniqueName: \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-kube-api-access-6cmv6\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.682565 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.682525    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-envoy\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.682565 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.682548    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-credential-socket\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.682715 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.682697    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-socket\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.682809 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.682794    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-podinfo\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.682873 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.682858    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-certs\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.784626 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.784593    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cmv6\" (UniqueName: \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-kube-api-access-6cmv6\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.784808 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.784634    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-envoy\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.784808 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.784742    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-credential-socket\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.784912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.784813    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-socket\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.784912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.784880    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-podinfo\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.785017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.784911    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-certs\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.785017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.784943    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7865f262-e4a2-4315-89c3-8e5655f1e47f-istiod-ca-cert\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.785017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.784964    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-data\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.785017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.785011    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-token\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.785237 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.785148    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-credential-socket\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.785297 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.785251    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-socket\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.785352 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.785312    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-certs\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.785395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.785365    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-data\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.785622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.785605    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7865f262-e4a2-4315-89c3-8e5655f1e47f-istiod-ca-cert\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.787194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.787170    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-envoy\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.787345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.787330    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-podinfo\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.802182 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.802150    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"]
Nov 01 04:17:15.810540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.810516    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-72snh"]
Nov 01 04:17:15.858540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.853030    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cmv6\" (UniqueName: \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-kube-api-access-6cmv6\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.862080 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.861253    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-token\") pod \"e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") " pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:15.901907 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:15.901866    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:16.324733 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:16.324701    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"]
Nov 01 04:17:16.326518 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:17:16.326487    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7865f262_e4a2_4315_89c3_8e5655f1e47f.slice/crio-95dc890b1d8643e23f08fbbf60fbcef7d05448afff42357c64052bb68b415eb7 WatchSource:0}: Error finding container 95dc890b1d8643e23f08fbbf60fbcef7d05448afff42357c64052bb68b415eb7: Status 404 returned error can't find the container with id 95dc890b1d8643e23f08fbbf60fbcef7d05448afff42357c64052bb68b415eb7
Nov 01 04:17:16.329081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:16.329046    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:16.329157 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:16.329114    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:16.329157 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:16.329141    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:16.606350 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:16.606237    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" event={"ID":"7865f262-e4a2-4315-89c3-8e5655f1e47f","Type":"ContainerStarted","Data":"30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e"}
Nov 01 04:17:16.606542 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:16.606355    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" event={"ID":"7865f262-e4a2-4315-89c3-8e5655f1e47f","Type":"ContainerStarted","Data":"95dc890b1d8643e23f08fbbf60fbcef7d05448afff42357c64052bb68b415eb7"}
Nov 01 04:17:16.719967 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:16.719908    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" podStartSLOduration=1.719892685 podStartE2EDuration="1.719892685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:16.717660742 +0000 UTC m=+2105.279693762" watchObservedRunningTime="2026-11-01 04:17:16.719892685 +0000 UTC m=+2105.281925701"
Nov 01 04:17:16.902544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:16.902459    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:17.601191 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:17.601156    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f715c6b5-edcb-46d9-8ed0-12c09b5164e7" path="/var/lib/kubelet/pods/f715c6b5-edcb-46d9-8ed0-12c09b5164e7/volumes"
Nov 01 04:17:17.908750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:17.908671    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:18.621766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:18.621731    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:18.622792 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:18.622767    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:17:22.273880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.273788    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"]
Nov 01 04:17:22.279065 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.279041    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.334731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.327805    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4c2a66-openshift-default-dockercfg-s68hw\""
Nov 01 04:17:22.377454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.376406    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-data\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.377454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.376500    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-podinfo\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.377454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.376541    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istiod-ca-cert\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.377454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.376585    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-envoy\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.377454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.376630    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-certs\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.377454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.376659    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-credential-socket\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.377454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.376699    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq99m\" (UniqueName: \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-kube-api-access-vq99m\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.377454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.376741    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-token\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.377454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.376788    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-socket\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.475363 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.475329    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"]
Nov 01 04:17:22.481703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.479698    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-data\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.481703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.479797    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-podinfo\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.481703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.479865    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istiod-ca-cert\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.481703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.479928    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-envoy\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.481703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.479989    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-certs\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.481703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.480020    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-credential-socket\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.481703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.480077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq99m\" (UniqueName: \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-kube-api-access-vq99m\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.481703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.480140    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-token\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.481703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.480210    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-socket\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.481703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.480955    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-socket\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.481703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.481295    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-data\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.483682 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.483657    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-certs\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.485024 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.484335    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istiod-ca-cert\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.485223 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.485036    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-credential-socket\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.489474 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.488576    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-envoy\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.489474 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.488945    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-podinfo\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.541588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.541510    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-token\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.541588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.541565    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq99m\" (UniqueName: \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-kube-api-access-vq99m\") pod \"e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") " pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:22.593650 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:22.593610    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:23.034291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:23.034263    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"]
Nov 01 04:17:23.035463 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:17:23.035421    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2984ced8_c4c5_44fa_b20c_6de34c0c891f.slice/crio-5b39a8e2b816f4ccd13f444e16720680590745014c3cb4e0040010d43b4253e5 WatchSource:0}: Error finding container 5b39a8e2b816f4ccd13f444e16720680590745014c3cb4e0040010d43b4253e5: Status 404 returned error can't find the container with id 5b39a8e2b816f4ccd13f444e16720680590745014c3cb4e0040010d43b4253e5
Nov 01 04:17:23.038204 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:23.038166    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:23.038288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:23.038238    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:23.038288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:23.038269    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:23.658855 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:23.658809    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" event={"ID":"2984ced8-c4c5-44fa-b20c-6de34c0c891f","Type":"ContainerStarted","Data":"aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9"}
Nov 01 04:17:23.659389 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:23.658878    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" event={"ID":"2984ced8-c4c5-44fa-b20c-6de34c0c891f","Type":"ContainerStarted","Data":"5b39a8e2b816f4ccd13f444e16720680590745014c3cb4e0040010d43b4253e5"}
Nov 01 04:17:23.774879 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:23.774820    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" podStartSLOduration=1.7748052140000001 podStartE2EDuration="1.774805214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:23.771183658 +0000 UTC m=+2112.333216675" watchObservedRunningTime="2026-11-01 04:17:23.774805214 +0000 UTC m=+2112.336838230"
Nov 01 04:17:24.045398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:24.045351    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"]
Nov 01 04:17:24.048534 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:24.048506    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc08a306-q7wcp"]
Nov 01 04:17:24.594630 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:24.594597    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:24.600696 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:24.600668    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:24.666762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:24.666723    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:24.667740 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:24.667719    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:17:25.600857 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:25.600810    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60799821-4071-47ef-b9dc-86189e2585cd" path="/var/lib/kubelet/pods/60799821-4071-47ef-b9dc-86189e2585cd/volumes"
Nov 01 04:17:25.754540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:25.754506    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"]
Nov 01 04:17:25.757817 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:25.757787    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d683e-z7wj5"]
Nov 01 04:17:27.604091 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:27.604034    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7802f48-af03-4377-929e-7685c26a964c" path="/var/lib/kubelet/pods/c7802f48-af03-4377-929e-7685c26a964c/volumes"
Nov 01 04:17:28.729995 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:17:28.729949    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:17:39.099591 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:39.099546    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:17:39.335273 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:39.099620    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:17:39.913211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:39.913171    4882 generic.go:356] "Generic (PLEG): container finished" podID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerID="516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b" exitCode=0
Nov 01 04:17:39.913445 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:39.913240    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" event={"ID":"5804f1e8-7bd2-443d-89ef-8bb1389845d1","Type":"ContainerDied","Data":"516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b"}
Nov 01 04:17:39.913792 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:39.913776    4882 scope.go:122] "RemoveContainer" containerID="516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b"
Nov 01 04:17:39.915719 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:39.915683    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:39.915826 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:39.915782    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:39.915893 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:39.915830    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:40.922467 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:40.922404    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" event={"ID":"5804f1e8-7bd2-443d-89ef-8bb1389845d1","Type":"ContainerStarted","Data":"f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393"}
Nov 01 04:17:40.924149 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:40.924119    4882 generic.go:356] "Generic (PLEG): container finished" podID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerID="03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827" exitCode=0
Nov 01 04:17:40.924149 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:40.924143    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" event={"ID":"769e10a8-d82e-4a7c-aec7-b79eb2cf1223","Type":"ContainerDied","Data":"03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827"}
Nov 01 04:17:40.924524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:40.924511    4882 scope.go:122] "RemoveContainer" containerID="03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827"
Nov 01 04:17:40.926037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:40.926008    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:40.926132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:40.926073    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:40.926132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:40.926106    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:17:41.470832 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:41.470795    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:17:41.487605 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:41.487567    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Nov 01 04:17:41.487886 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:41.487854    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Nov 01 04:17:41.932665 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:41.932574    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" event={"ID":"769e10a8-d82e-4a7c-aec7-b79eb2cf1223","Type":"ContainerStarted","Data":"96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26"}
Nov 01 04:17:42.146536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:42.146496    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:17:42.146825 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:42.146798    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 04:17:42.146890 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:42.146851    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 04:17:42.470894 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:42.470849    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Nov 01 04:17:42.471218 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:42.471186    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Nov 01 04:17:43.146419 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:43.146378    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 04:17:43.146892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:43.146471    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 04:17:43.471464 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:43.470877    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Nov 01 04:17:43.471464 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:43.470940    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Nov 01 04:17:44.158054 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:44.158017    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 04:17:44.159612 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:44.159581    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 04:17:44.484920 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:44.484875    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Nov 01 04:17:44.485360 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:44.485323    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Nov 01 04:17:45.146583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:45.146540    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 04:17:45.146789 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:45.146607    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 04:17:45.470883 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:45.470841    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Nov 01 04:17:45.471535 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:45.471497    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Nov 01 04:17:45.856380 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:45.856294    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:17:46.147167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:46.147057    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 04:17:46.147167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:46.147117    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 04:17:46.490982 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:46.490944    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:17:46.492076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:46.492055    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:17:47.146273 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:47.146235    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 04:17:47.146529 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:47.146296    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 04:17:48.150782 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:48.150750    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:17:48.151247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:48.151208    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:17:48.151951 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:17:48.151935    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:17:58.723744 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:17:58.723699    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:18:13.262478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:13.262409    4882 scope.go:122] "RemoveContainer" containerID="5eac3fea8fa0c52e0eea0bb473f79101baecf60590c07bce8208c45f2a61a3da"
Nov 01 04:18:24.565144 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.565105    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"]
Nov 01 04:18:24.573933 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.573902    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.640451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.631510    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0ee0f8-openshift-default-dockercfg-qnvcj\""
Nov 01 04:18:24.693453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.678728    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-envoy\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.713264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.695191    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-socket\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.714352 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.714313    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-token\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.714655 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.714629    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-certs\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.714823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.714802    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb7nb\" (UniqueName: \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-kube-api-access-nb7nb\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.715002 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.714981    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-data\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.717402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.715184    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-podinfo\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.717601 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.717578    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3109f72-8937-4f69-b7f0-25b2e4870336-istiod-ca-cert\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.717942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.717886    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-credential-socket\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.732570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.732529    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"]
Nov 01 04:18:24.819252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819161    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-envoy\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.819252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819209    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-socket\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.819252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819243    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-token\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.819575 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819309    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-certs\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.819575 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819345    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb7nb\" (UniqueName: \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-kube-api-access-nb7nb\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.819575 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819378    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-data\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.819750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819616    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-podinfo\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.819750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819656    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-socket\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.819750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819671    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3109f72-8937-4f69-b7f0-25b2e4870336-istiod-ca-cert\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.819750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819718    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-data\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.819750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819744    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-certs\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.820024 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.819840    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-credential-socket\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.820158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.820130    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-credential-socket\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.820302 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.820254    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3109f72-8937-4f69-b7f0-25b2e4870336-istiod-ca-cert\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.821904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.821885    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-envoy\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.822173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.822152    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-podinfo\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.878681 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.878646    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-token\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.878864 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.878712    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb7nb\" (UniqueName: \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-kube-api-access-nb7nb\") pod \"e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") " pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:24.888379 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:24.888346    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:25.309084 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:25.309052    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"]
Nov 01 04:18:25.311219 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:18:25.311192    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3109f72_8937_4f69_b7f0_25b2e4870336.slice/crio-0caff8734f68cbe2c379b8f05c79d6846c06bad229cab14605c4c72f85545930 WatchSource:0}: Error finding container 0caff8734f68cbe2c379b8f05c79d6846c06bad229cab14605c4c72f85545930: Status 404 returned error can't find the container with id 0caff8734f68cbe2c379b8f05c79d6846c06bad229cab14605c4c72f85545930
Nov 01 04:18:25.313832 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:25.313795    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:18:25.313910 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:25.313871    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:18:25.313910 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:25.313902    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:18:26.250037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:26.249993    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" event={"ID":"c3109f72-8937-4f69-b7f0-25b2e4870336","Type":"ContainerStarted","Data":"80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa"}
Nov 01 04:18:26.250521 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:26.250060    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" event={"ID":"c3109f72-8937-4f69-b7f0-25b2e4870336","Type":"ContainerStarted","Data":"0caff8734f68cbe2c379b8f05c79d6846c06bad229cab14605c4c72f85545930"}
Nov 01 04:18:26.411258 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:26.411192    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" podStartSLOduration=2.411177242 podStartE2EDuration="2.411177242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:18:26.410040773 +0000 UTC m=+2174.972073789" watchObservedRunningTime="2026-11-01 04:18:26.411177242 +0000 UTC m=+2174.973210258"
Nov 01 04:18:26.888610 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:26.888561    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:26.896282 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:26.896241    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:27.257736 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:27.257700    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:27.258938 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:27.258911    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:18:28.724290 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:18:28.724254    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:18:53.459736 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:53.459701    4882 generic.go:356] "Generic (PLEG): container finished" podID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerID="30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e" exitCode=0
Nov 01 04:18:53.460307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:53.459751    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" event={"ID":"7865f262-e4a2-4315-89c3-8e5655f1e47f","Type":"ContainerDied","Data":"30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e"}
Nov 01 04:18:53.460307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:53.460279    4882 scope.go:122] "RemoveContainer" containerID="30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e"
Nov 01 04:18:53.461971 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:53.461942    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:18:53.462065 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:53.462020    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:18:53.462065 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:53.462055    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:18:54.491352 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:54.491310    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" event={"ID":"7865f262-e4a2-4315-89c3-8e5655f1e47f","Type":"ContainerStarted","Data":"5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d"}
Nov 01 04:18:54.902897 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:54.902800    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:18:54.903093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:54.903049    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 04:18:54.903178 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:54.903107    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 04:18:55.902384 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:55.902347    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 04:18:55.902928 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:55.902399    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 04:18:56.902766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:56.902732    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 04:18:56.903183 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:56.902790    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 04:18:57.903314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:57.903273    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 04:18:57.903777 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:57.903330    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 04:18:58.724866 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:18:58.724827    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:18:58.903087 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:58.903047    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 04:18:58.903284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:58.903115    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 04:18:59.908317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:59.908282    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:18:59.908752 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:59.908714    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:18:59.909659 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:18:59.909638    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:19:05.045874 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:05.045783    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"]
Nov 01 04:19:05.091883 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:05.091847    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a45508-5fx2d"]
Nov 01 04:19:05.600170 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:05.600129    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="773694d7-d894-490f-be9a-0a064f68d0db" path="/var/lib/kubelet/pods/773694d7-d894-490f-be9a-0a064f68d0db/volumes"
Nov 01 04:19:13.293927 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:13.293881    4882 scope.go:122] "RemoveContainer" containerID="50ff95a5b8151488c39f38f9a8f494b5dfa99752a053ac60d7eb702ff64f4c63"
Nov 01 04:19:28.723635 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:19:28.723580    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:19:30.508454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.508408    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"]
Nov 01 04:19:30.514052 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.514022    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.514834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.514804    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skbbs\" (UniqueName: \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-kube-api-access-skbbs\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.514953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.514934    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istiod-ca-cert\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.515021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.514979    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-podinfo\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.515129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.515108    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-token\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.515194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.515145    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-envoy\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.515240 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.515197    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-socket\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.515240 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.515231    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-certs\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.515327 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.515284    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-credential-socket\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.515392 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.515349    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-data\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.565886 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.565853    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0882f4-openshift-default-dockercfg-sq49m\""
Nov 01 04:19:30.687634 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.687599    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-credential-socket\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.696021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.695983    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-credential-socket\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.696700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.696673    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-data\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.700556 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.687894    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-data\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.700875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.700851    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skbbs\" (UniqueName: \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-kube-api-access-skbbs\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.701226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.701200    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istiod-ca-cert\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.709579 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.709543    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-podinfo\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.709934 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.709909    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-token\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.716022 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.715994    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-envoy\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.716396 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.716281    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-socket\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.717772 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.717379    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-podinfo\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.718827 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.717640    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-certs\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.719143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.718822    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-certs\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.728570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.704057    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istiod-ca-cert\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.728570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.724743    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-socket\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.729009 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.728986    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"]
Nov 01 04:19:30.732551 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.732505    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-envoy\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.760546 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.760478    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skbbs\" (UniqueName: \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-kube-api-access-skbbs\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.778920 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.778891    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-token\") pod \"e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") " pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:30.837151 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:30.837111    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:31.301521 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:31.301491    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"]
Nov 01 04:19:31.303140 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:19:31.303111    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5256b4_e61e_4ac2_82b5_9f28bc217d0b.slice/crio-ff74d15f2d396add0bb8d660e2cb3147ba752a5d2d3ca7ce6d7fdd81c707e289 WatchSource:0}: Error finding container ff74d15f2d396add0bb8d660e2cb3147ba752a5d2d3ca7ce6d7fdd81c707e289: Status 404 returned error can't find the container with id ff74d15f2d396add0bb8d660e2cb3147ba752a5d2d3ca7ce6d7fdd81c707e289
Nov 01 04:19:31.305805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:31.305772    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:19:31.305900 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:31.305853    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:19:31.305900 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:31.305885    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:19:31.752911 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:31.752866    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" event={"ID":"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b","Type":"ContainerStarted","Data":"28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181"}
Nov 01 04:19:31.753397 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:31.752959    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" event={"ID":"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b","Type":"ContainerStarted","Data":"ff74d15f2d396add0bb8d660e2cb3147ba752a5d2d3ca7ce6d7fdd81c707e289"}
Nov 01 04:19:31.839316 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:31.839275    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:31.868585 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:31.868516    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" podStartSLOduration=1.868502041 podStartE2EDuration="1.868502041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:19:31.865647064 +0000 UTC m=+2240.427680081" watchObservedRunningTime="2026-11-01 04:19:31.868502041 +0000 UTC m=+2240.430535058"
Nov 01 04:19:32.844996 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:32.844965    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:33.768164 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:33.768132    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:33.769220 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:33.769191    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:19:35.253896 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.253855    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"]
Nov 01 04:19:35.258237 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.258214    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:19:35.354514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.354423    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-601a4b-dockercfg-hzkvz\""
Nov 01 04:19:35.354756 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.354736    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-601a4b\""
Nov 01 04:19:35.358064 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.358041    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"]
Nov 01 04:19:35.386440 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.386399    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dw6b\" (UniqueName: \"kubernetes.io/projected/9251b815-0f89-4f7c-bba6-046d34815b35-kube-api-access-4dw6b\") pod \"payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw\" (UID: \"9251b815-0f89-4f7c-bba6-046d34815b35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:19:35.386685 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.386660    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9251b815-0f89-4f7c-bba6-046d34815b35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw\" (UID: \"9251b815-0f89-4f7c-bba6-046d34815b35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:19:35.498078 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.497596    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dw6b\" (UniqueName: \"kubernetes.io/projected/9251b815-0f89-4f7c-bba6-046d34815b35-kube-api-access-4dw6b\") pod \"payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw\" (UID: \"9251b815-0f89-4f7c-bba6-046d34815b35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:19:35.498078 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.497822    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9251b815-0f89-4f7c-bba6-046d34815b35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw\" (UID: \"9251b815-0f89-4f7c-bba6-046d34815b35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:19:35.503750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.503705    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9251b815-0f89-4f7c-bba6-046d34815b35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw\" (UID: \"9251b815-0f89-4f7c-bba6-046d34815b35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:19:35.558118 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.558042    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dw6b\" (UniqueName: \"kubernetes.io/projected/9251b815-0f89-4f7c-bba6-046d34815b35-kube-api-access-4dw6b\") pod \"payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw\" (UID: \"9251b815-0f89-4f7c-bba6-046d34815b35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:19:35.572936 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.572908    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:19:35.960546 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:35.960519    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"]
Nov 01 04:19:35.962366 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:19:35.962325    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9251b815_0f89_4f7c_bba6_046d34815b35.slice/crio-118196d58da56b5f2ca031e447db3767b0c4422227c1b1dcd103d94904b15c69 WatchSource:0}: Error finding container 118196d58da56b5f2ca031e447db3767b0c4422227c1b1dcd103d94904b15c69: Status 404 returned error can't find the container with id 118196d58da56b5f2ca031e447db3767b0c4422227c1b1dcd103d94904b15c69
Nov 01 04:19:36.787278 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:36.787242    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw" event={"ID":"9251b815-0f89-4f7c-bba6-046d34815b35","Type":"ContainerStarted","Data":"60128db4946542f534e6c5b80028c7d9ddd415141433c913f912510435961ca7"}
Nov 01 04:19:36.787717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:36.787299    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw" event={"ID":"9251b815-0f89-4f7c-bba6-046d34815b35","Type":"ContainerStarted","Data":"118196d58da56b5f2ca031e447db3767b0c4422227c1b1dcd103d94904b15c69"}
Nov 01 04:19:36.787717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:36.787451    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:19:36.897340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:36.897245    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw" podStartSLOduration=1.897228522 podStartE2EDuration="1.897228522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:19:36.896574728 +0000 UTC m=+2245.458607745" watchObservedRunningTime="2026-11-01 04:19:36.897228522 +0000 UTC m=+2245.459261542"
Nov 01 04:19:47.796648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:19:47.796613    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:19:58.723022 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:19:58.722982    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:20:01.988644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:01.988606    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"]
Nov 01 04:20:01.993727 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:01.993704    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.049456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.048524    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-2afc0d-openshift-default-dockercfg-6mwd2\""
Nov 01 04:20:02.124377 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.121309    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"]
Nov 01 04:20:02.134920 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.134863    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-credential-socket\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.135241 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.135207    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-podinfo\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.135457 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.135438    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-certs\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.135633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.135615    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-token\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.135766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.135750    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istiod-ca-cert\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.135923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.135905    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9c7p\" (UniqueName: \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-kube-api-access-m9c7p\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.136051 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.136037    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-envoy\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.136201 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.136188    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-data\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.136318 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.136302    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-socket\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.238269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.238132    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-envoy\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.238686 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.238615    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-data\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.238871 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.238851    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-socket\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.239079 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.239059    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-credential-socket\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.240036 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.240008    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-podinfo\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.240134 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.240105    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-certs\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.240205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.240180    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-token\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.240271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.240226    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istiod-ca-cert\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.240337 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.240297    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9c7p\" (UniqueName: \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-kube-api-access-m9c7p\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.240714 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.239349    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-data\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.241061 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.239780    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-credential-socket\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.241360 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.239683    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-socket\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.241480 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.241380    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-certs\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.242046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.242024    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istiod-ca-cert\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.243237 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.243164    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-envoy\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.243954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.243939    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-podinfo\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.297472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.297419    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9c7p\" (UniqueName: \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-kube-api-access-m9c7p\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.300595 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.300564    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-token\") pod \"e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") " pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.350556 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.350510    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:02.808478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.808403    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"]
Nov 01 04:20:02.810064 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:20:02.810040    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50a6fa4_73bc_42b8_a250_6f12d7e2bf9b.slice/crio-228578076ad23a7fa95efaac0d20ddce8f02a5d6977fdaee1cae2592395d2314 WatchSource:0}: Error finding container 228578076ad23a7fa95efaac0d20ddce8f02a5d6977fdaee1cae2592395d2314: Status 404 returned error can't find the container with id 228578076ad23a7fa95efaac0d20ddce8f02a5d6977fdaee1cae2592395d2314
Nov 01 04:20:02.812975 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.812938    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:02.813050 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.813025    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:02.813089 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:02.813071    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:03.208711 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.208675    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" event={"ID":"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b","Type":"ContainerStarted","Data":"3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee"}
Nov 01 04:20:03.209088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.208742    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" event={"ID":"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b","Type":"ContainerStarted","Data":"228578076ad23a7fa95efaac0d20ddce8f02a5d6977fdaee1cae2592395d2314"}
Nov 01 04:20:03.321715 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.321645    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" podStartSLOduration=2.321626556 podStartE2EDuration="2.321626556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:20:03.320492922 +0000 UTC m=+2271.882525940" watchObservedRunningTime="2026-11-01 04:20:03.321626556 +0000 UTC m=+2271.883659574"
Nov 01 04:20:03.351192 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.351156    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:03.357864 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.357827    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:03.487245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.486214    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"]
Nov 01 04:20:03.487245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.486555    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerName="istio-proxy" containerID="cri-o://21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2" gracePeriod=30
Nov 01 04:20:03.492793 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.492761    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"]
Nov 01 04:20:03.493119 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.493050    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" containerName="istio-proxy" containerID="cri-o://ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3" gracePeriod=30
Nov 01 04:20:03.525537 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.525500    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"]
Nov 01 04:20:03.525772 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.525744    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" podUID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" containerName="istio-proxy" containerID="cri-o://7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0" gracePeriod=30
Nov 01 04:20:03.531725 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.531694    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"]
Nov 01 04:20:03.531984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:03.531935    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy" containerID="cri-o://9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434" gracePeriod=30
Nov 01 04:20:04.215797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:04.215754    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:04.216699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:04.216674    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:20:04.421363 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:04.421319    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-flvv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:20:04.421577 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:04.421393    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:20:06.235988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:06.235956    4882 generic.go:356] "Generic (PLEG): container finished" podID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerID="aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9" exitCode=0
Nov 01 04:20:06.236411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:06.236032    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" event={"ID":"2984ced8-c4c5-44fa-b20c-6de34c0c891f","Type":"ContainerDied","Data":"aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9"}
Nov 01 04:20:06.236567 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:06.236543    4882 scope.go:122] "RemoveContainer" containerID="aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9"
Nov 01 04:20:06.238038 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:06.238013    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:06.238110 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:06.238078    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:06.238146 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:06.238112    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:07.300449 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:07.291115    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" event={"ID":"2984ced8-c4c5-44fa-b20c-6de34c0c891f","Type":"ContainerStarted","Data":"ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7"}
Nov 01 04:20:07.605320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:07.605222    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:20:07.608578 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:07.608538    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 01 04:20:07.608742 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:07.608644    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 01 04:20:08.627674 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:08.612830    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 01 04:20:08.627674 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:08.612885    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 01 04:20:09.022698 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.022667    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:20:09.054159 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.054130    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:20:09.125996 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.125909    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-envoy\") pod \"a4921413-f98d-4883-a532-a4e1c45219d7\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") "
Nov 01 04:20:09.126607 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.126586    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-certs\") pod \"a4921413-f98d-4883-a532-a4e1c45219d7\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") "
Nov 01 04:20:09.127617 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.126959    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-certs" pod "a4921413-f98d-4883-a532-a4e1c45219d7" (UID: "a4921413-f98d-4883-a532-a4e1c45219d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.128077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.127832    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-socket\") pod \"1fbfedcc-725a-4408-883b-62a2c49e655e\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") "
Nov 01 04:20:09.128148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.128076    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-socket" pod "1fbfedcc-725a-4408-883b-62a2c49e655e" (UID: "1fbfedcc-725a-4408-883b-62a2c49e655e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.128148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.128138    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-istio-token\" (UniqueName: \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-istio-token\") pod \"a4921413-f98d-4883-a532-a4e1c45219d7\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") "
Nov 01 04:20:09.128576 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.128540    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4921413-f98d-4883-a532-a4e1c45219d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4921413-f98d-4883-a532-a4e1c45219d7-istiod-ca-cert\") pod \"a4921413-f98d-4883-a532-a4e1c45219d7\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") "
Nov 01 04:20:09.128689 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.128618    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-data\") pod \"1fbfedcc-725a-4408-883b-62a2c49e655e\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") "
Nov 01 04:20:09.128689 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.128657    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-token\" (UniqueName: \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-token\") pod \"1fbfedcc-725a-4408-883b-62a2c49e655e\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") "
Nov 01 04:20:09.128811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.128696    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fbfedcc-725a-4408-883b-62a2c49e655e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fbfedcc-725a-4408-883b-62a2c49e655e-istiod-ca-cert\") pod \"1fbfedcc-725a-4408-883b-62a2c49e655e\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") "
Nov 01 04:20:09.128902 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.128880    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-credential-socket\") pod \"a4921413-f98d-4883-a532-a4e1c45219d7\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") "
Nov 01 04:20:09.128976 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.128961    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-podinfo\") pod \"1fbfedcc-725a-4408-883b-62a2c49e655e\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") "
Nov 01 04:20:09.129037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.128996    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-envoy\") pod \"1fbfedcc-725a-4408-883b-62a2c49e655e\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") "
Nov 01 04:20:09.129037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.129026    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4921413-f98d-4883-a532-a4e1c45219d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4921413-f98d-4883-a532-a4e1c45219d7-istio-podinfo\") pod \"a4921413-f98d-4883-a532-a4e1c45219d7\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") "
Nov 01 04:20:09.129255 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.129069    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-kube-api-access-fftkv\" (UniqueName: \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-kube-api-access-fftkv\") pod \"1fbfedcc-725a-4408-883b-62a2c49e655e\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") "
Nov 01 04:20:09.129255 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.129105    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-kube-api-access-mwk26\" (UniqueName: \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-kube-api-access-mwk26\") pod \"a4921413-f98d-4883-a532-a4e1c45219d7\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") "
Nov 01 04:20:09.129255 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.129142    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-credential-socket\") pod \"1fbfedcc-725a-4408-883b-62a2c49e655e\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") "
Nov 01 04:20:09.129255 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.129180    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-data" pod "1fbfedcc-725a-4408-883b-62a2c49e655e" (UID: "1fbfedcc-725a-4408-883b-62a2c49e655e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.129608 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.129293    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4921413-f98d-4883-a532-a4e1c45219d7-istiod-ca-cert" pod "a4921413-f98d-4883-a532-a4e1c45219d7" (UID: "a4921413-f98d-4883-a532-a4e1c45219d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:20:09.129608 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.129483    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-credential-socket" pod "a4921413-f98d-4883-a532-a4e1c45219d7" (UID: "a4921413-f98d-4883-a532-a4e1c45219d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.129929 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.129906    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-socket\") pod \"a4921413-f98d-4883-a532-a4e1c45219d7\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") "
Nov 01 04:20:09.130024 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.129960    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-data\") pod \"a4921413-f98d-4883-a532-a4e1c45219d7\" (UID: \"a4921413-f98d-4883-a532-a4e1c45219d7\") "
Nov 01 04:20:09.130024 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130004    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-certs\") pod \"1fbfedcc-725a-4408-883b-62a2c49e655e\" (UID: \"1fbfedcc-725a-4408-883b-62a2c49e655e\") "
Nov 01 04:20:09.130132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130085    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-credential-socket" pod "1fbfedcc-725a-4408-883b-62a2c49e655e" (UID: "1fbfedcc-725a-4408-883b-62a2c49e655e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.130317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130295    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-socket" pod "a4921413-f98d-4883-a532-a4e1c45219d7" (UID: "a4921413-f98d-4883-a532-a4e1c45219d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.130492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130471    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fbfedcc-725a-4408-883b-62a2c49e655e-istiod-ca-cert" pod "1fbfedcc-725a-4408-883b-62a2c49e655e" (UID: "1fbfedcc-725a-4408-883b-62a2c49e655e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:20:09.130783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130757    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-data" pod "a4921413-f98d-4883-a532-a4e1c45219d7" (UID: "a4921413-f98d-4883-a532-a4e1c45219d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.130880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130831    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.130880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130852    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.130880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130867    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.131047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130882    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.131047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130896    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.131047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130911    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4921413-f98d-4883-a532-a4e1c45219d7-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.131047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130926    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.131047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130941    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fbfedcc-725a-4408-883b-62a2c49e655e-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.131047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130961    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-certs" pod "1fbfedcc-725a-4408-883b-62a2c49e655e" (UID: "1fbfedcc-725a-4408-883b-62a2c49e655e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.131047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.130960    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-envoy" pod "a4921413-f98d-4883-a532-a4e1c45219d7" (UID: "a4921413-f98d-4883-a532-a4e1c45219d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.132695 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.132593    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-podinfo" pod "1fbfedcc-725a-4408-883b-62a2c49e655e" (UID: "1fbfedcc-725a-4408-883b-62a2c49e655e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:20:09.132695 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.132664    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4921413-f98d-4883-a532-a4e1c45219d7-istio-podinfo" pod "a4921413-f98d-4883-a532-a4e1c45219d7" (UID: "a4921413-f98d-4883-a532-a4e1c45219d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:20:09.133233 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.133181    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-envoy" pod "1fbfedcc-725a-4408-883b-62a2c49e655e" (UID: "1fbfedcc-725a-4408-883b-62a2c49e655e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.133897 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.133871    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-kube-api-access-fftkv" pod "1fbfedcc-725a-4408-883b-62a2c49e655e" (UID: "1fbfedcc-725a-4408-883b-62a2c49e655e"). InnerVolumeSpecName "kube-api-access-fftkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:20:09.134019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.133914    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-kube-api-access-mwk26" pod "a4921413-f98d-4883-a532-a4e1c45219d7" (UID: "a4921413-f98d-4883-a532-a4e1c45219d7"). InnerVolumeSpecName "kube-api-access-mwk26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:20:09.134184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.134157    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-istio-token" pod "a4921413-f98d-4883-a532-a4e1c45219d7" (UID: "a4921413-f98d-4883-a532-a4e1c45219d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:20:09.134661 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.134639    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-token" pod "1fbfedcc-725a-4408-883b-62a2c49e655e" (UID: "1fbfedcc-725a-4408-883b-62a2c49e655e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:20:09.142716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.142697    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 04:20:09.144649 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.144632    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 04:20:09.232301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232263    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-token\" (UniqueName: \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-token\") pod \"86c0d040-6d16-4890-9595-43e09ef4f0b0\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") "
Nov 01 04:20:09.232492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232345    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-data\") pod \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") "
Nov 01 04:20:09.232492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232394    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-certs\") pod \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") "
Nov 01 04:20:09.232492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232483    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-certs\") pod \"86c0d040-6d16-4890-9595-43e09ef4f0b0\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") "
Nov 01 04:20:09.232646 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232512    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-data\") pod \"86c0d040-6d16-4890-9595-43e09ef4f0b0\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") "
Nov 01 04:20:09.232646 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232556    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-token\" (UniqueName: \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-token\") pod \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") "
Nov 01 04:20:09.232646 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232592    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-socket\") pod \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") "
Nov 01 04:20:09.232747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232656    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-credential-socket\") pod \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") "
Nov 01 04:20:09.232747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232709    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-envoy\") pod \"86c0d040-6d16-4890-9595-43e09ef4f0b0\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") "
Nov 01 04:20:09.232747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232735    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c0d040-6d16-4890-9595-43e09ef4f0b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c0d040-6d16-4890-9595-43e09ef4f0b0-istiod-ca-cert\") pod \"86c0d040-6d16-4890-9595-43e09ef4f0b0\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") "
Nov 01 04:20:09.232841 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232786    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-envoy\") pod \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") "
Nov 01 04:20:09.232841 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232817    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istiod-ca-cert\") pod \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") "
Nov 01 04:20:09.232899 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232874    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-podinfo\") pod \"86c0d040-6d16-4890-9595-43e09ef4f0b0\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") "
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232936    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-kube-api-access-fwgdz\" (UniqueName: \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-kube-api-access-fwgdz\") pod \"86c0d040-6d16-4890-9595-43e09ef4f0b0\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") "
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.232969    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-kube-api-access-9v86t\" (UniqueName: \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-kube-api-access-9v86t\") pod \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") "
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233013    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-socket\") pod \"86c0d040-6d16-4890-9595-43e09ef4f0b0\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") "
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233058    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-credential-socket\") pod \"86c0d040-6d16-4890-9595-43e09ef4f0b0\" (UID: \"86c0d040-6d16-4890-9595-43e09ef4f0b0\") "
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233167    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-podinfo\") pod \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\" (UID: \"821ecb15-ab45-430e-8fc7-b7b1d9b069fd\") "
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233235    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c0d040-6d16-4890-9595-43e09ef4f0b0-istiod-ca-cert" pod "86c0d040-6d16-4890-9595-43e09ef4f0b0" (UID: "86c0d040-6d16-4890-9595-43e09ef4f0b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233514    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-socket" pod "821ecb15-ab45-430e-8fc7-b7b1d9b069fd" (UID: "821ecb15-ab45-430e-8fc7-b7b1d9b069fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233849    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233871    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233870    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-credential-socket" pod "821ecb15-ab45-430e-8fc7-b7b1d9b069fd" (UID: "821ecb15-ab45-430e-8fc7-b7b1d9b069fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233887    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233905    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233921    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4921413-f98d-4883-a532-a4e1c45219d7-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233935    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fftkv\" (UniqueName: \"kubernetes.io/projected/1fbfedcc-725a-4408-883b-62a2c49e655e-kube-api-access-fftkv\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233952    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwk26\" (UniqueName: \"kubernetes.io/projected/a4921413-f98d-4883-a532-a4e1c45219d7-kube-api-access-mwk26\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233969    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233985    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.234000    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fbfedcc-725a-4408-883b-62a2c49e655e-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.234014    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c0d040-6d16-4890-9595-43e09ef4f0b0-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.234027    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4921413-f98d-4883-a532-a4e1c45219d7-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.233903    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-certs" pod "821ecb15-ab45-430e-8fc7-b7b1d9b069fd" (UID: "821ecb15-ab45-430e-8fc7-b7b1d9b069fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.234092    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-socket" pod "86c0d040-6d16-4890-9595-43e09ef4f0b0" (UID: "86c0d040-6d16-4890-9595-43e09ef4f0b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.234236    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-certs" pod "86c0d040-6d16-4890-9595-43e09ef4f0b0" (UID: "86c0d040-6d16-4890-9595-43e09ef4f0b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.234699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.234420    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-data" pod "86c0d040-6d16-4890-9595-43e09ef4f0b0" (UID: "86c0d040-6d16-4890-9595-43e09ef4f0b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.236716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.236343    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istiod-ca-cert" pod "821ecb15-ab45-430e-8fc7-b7b1d9b069fd" (UID: "821ecb15-ab45-430e-8fc7-b7b1d9b069fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:20:09.236716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.236456    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-data" pod "821ecb15-ab45-430e-8fc7-b7b1d9b069fd" (UID: "821ecb15-ab45-430e-8fc7-b7b1d9b069fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.236892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.236723    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-credential-socket" pod "86c0d040-6d16-4890-9595-43e09ef4f0b0" (UID: "86c0d040-6d16-4890-9595-43e09ef4f0b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.237194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.237025    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-token" pod "821ecb15-ab45-430e-8fc7-b7b1d9b069fd" (UID: "821ecb15-ab45-430e-8fc7-b7b1d9b069fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:20:09.237194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.237111    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-podinfo" pod "86c0d040-6d16-4890-9595-43e09ef4f0b0" (UID: "86c0d040-6d16-4890-9595-43e09ef4f0b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:20:09.237194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.237153    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-kube-api-access-fwgdz" pod "86c0d040-6d16-4890-9595-43e09ef4f0b0" (UID: "86c0d040-6d16-4890-9595-43e09ef4f0b0"). InnerVolumeSpecName "kube-api-access-fwgdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:20:09.237194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.237181    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-kube-api-access-9v86t" pod "821ecb15-ab45-430e-8fc7-b7b1d9b069fd" (UID: "821ecb15-ab45-430e-8fc7-b7b1d9b069fd"). InnerVolumeSpecName "kube-api-access-9v86t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:20:09.237471 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.237223    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-token" pod "86c0d040-6d16-4890-9595-43e09ef4f0b0" (UID: "86c0d040-6d16-4890-9595-43e09ef4f0b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:20:09.237636 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.237613    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-podinfo" pod "821ecb15-ab45-430e-8fc7-b7b1d9b069fd" (UID: "821ecb15-ab45-430e-8fc7-b7b1d9b069fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:20:09.237850 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.237829    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-envoy" pod "86c0d040-6d16-4890-9595-43e09ef4f0b0" (UID: "86c0d040-6d16-4890-9595-43e09ef4f0b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.238831 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.238810    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-envoy" pod "821ecb15-ab45-430e-8fc7-b7b1d9b069fd" (UID: "821ecb15-ab45-430e-8fc7-b7b1d9b069fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:20:09.329304 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.329267    4882 generic.go:356] "Generic (PLEG): container finished" podID="a4921413-f98d-4883-a532-a4e1c45219d7" containerID="ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3" exitCode=0
Nov 01 04:20:09.329515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.329321    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"
Nov 01 04:20:09.329515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.329356    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" event={"ID":"a4921413-f98d-4883-a532-a4e1c45219d7","Type":"ContainerDied","Data":"ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3"}
Nov 01 04:20:09.329515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.329477    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt" event={"ID":"a4921413-f98d-4883-a532-a4e1c45219d7","Type":"ContainerDied","Data":"6834917fc4ba04e6d1060c8e557cdae40d1fcac704577b69d234fd27bbec3eb9"}
Nov 01 04:20:09.329515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.329514    4882 scope.go:122] "RemoveContainer" containerID="ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3"
Nov 01 04:20:09.333504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.330985    4882 generic.go:356] "Generic (PLEG): container finished" podID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerID="9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434" exitCode=0
Nov 01 04:20:09.333504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.331059    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" event={"ID":"86c0d040-6d16-4890-9595-43e09ef4f0b0","Type":"ContainerDied","Data":"9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434"}
Nov 01 04:20:09.333504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.331091    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6" event={"ID":"86c0d040-6d16-4890-9595-43e09ef4f0b0","Type":"ContainerDied","Data":"cf903601997c0b54b462092e1767b72ee66ae48f290f35c2f0e2d78506a00918"}
Nov 01 04:20:09.333504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.331150    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.334890    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.334913    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.334924    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.334934    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.334948    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.334963    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.334972    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.334981    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.334993    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.335004    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.335013    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c0d040-6d16-4890-9595-43e09ef4f0b0-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.335022    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwgdz\" (UniqueName: \"kubernetes.io/projected/86c0d040-6d16-4890-9595-43e09ef4f0b0-kube-api-access-fwgdz\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.335035    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v86t\" (UniqueName: \"kubernetes.io/projected/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-kube-api-access-9v86t\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.335035    4882 generic.go:356] "Generic (PLEG): container finished" podID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" containerID="7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0" exitCode=0
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.335048    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.335061    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c0d040-6d16-4890-9595-43e09ef4f0b0-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.335074    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821ecb15-ab45-430e-8fc7-b7b1d9b069fd-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.335108    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" event={"ID":"821ecb15-ab45-430e-8fc7-b7b1d9b069fd","Type":"ContainerDied","Data":"7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0"}
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.335147    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565" event={"ID":"821ecb15-ab45-430e-8fc7-b7b1d9b069fd","Type":"ContainerDied","Data":"c505e7452399efcd5a3928cfedf398ebf93525beba102ff693495a6aee559f08"}
Nov 01 04:20:09.336288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.335186    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"
Nov 01 04:20:09.340402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.338216    4882 generic.go:356] "Generic (PLEG): container finished" podID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerID="21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2" exitCode=0
Nov 01 04:20:09.340402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.338334    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" event={"ID":"1fbfedcc-725a-4408-883b-62a2c49e655e","Type":"ContainerDied","Data":"21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2"}
Nov 01 04:20:09.340402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.338361    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5" event={"ID":"1fbfedcc-725a-4408-883b-62a2c49e655e","Type":"ContainerDied","Data":"8f2ee0c3a0f08dcb66276769ee284e6076db9ebd15be683a7dce12507884e0c3"}
Nov 01 04:20:09.340402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.338407    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"
Nov 01 04:20:09.346364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.343514    4882 scope.go:122] "RemoveContainer" containerID="ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3"
Nov 01 04:20:09.346364 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:20:09.343774    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3\": container with ID starting with ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3 not found: ID does not exist" containerID="ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3"
Nov 01 04:20:09.346364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.343795    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3"} err="failed to get container status \"ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3\": rpc error: code = NotFound desc = could not find container \"ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3\": container with ID starting with ce33413d417cf2d504fa5c13269321aac86a85ec80b52576a2b1c480066362e3 not found: ID does not exist"
Nov 01 04:20:09.346364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.343813    4882 scope.go:122] "RemoveContainer" containerID="9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434"
Nov 01 04:20:09.361371 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.361354    4882 scope.go:122] "RemoveContainer" containerID="9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434"
Nov 01 04:20:09.418683 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:20:09.416632    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434\": container with ID starting with 9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434 not found: ID does not exist" containerID="9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434"
Nov 01 04:20:09.418683 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.416682    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434"} err="failed to get container status \"9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434\": rpc error: code = NotFound desc = could not find container \"9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434\": container with ID starting with 9870ca89a4c5d33b65222b5885fc5848952f63078dd11f19182083706de0e434 not found: ID does not exist"
Nov 01 04:20:09.418683 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.416715    4882 scope.go:122] "RemoveContainer" containerID="7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0"
Nov 01 04:20:09.435446 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.435266    4882 scope.go:122] "RemoveContainer" containerID="7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0"
Nov 01 04:20:09.436015 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:20:09.435828    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0\": container with ID starting with 7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0 not found: ID does not exist" containerID="7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0"
Nov 01 04:20:09.436015 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.435890    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0"} err="failed to get container status \"7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0\": rpc error: code = NotFound desc = could not find container \"7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0\": container with ID starting with 7e57419017e7b7a1e407a891269a9375b477261acf0549955760ded5e22da8b0 not found: ID does not exist"
Nov 01 04:20:09.436015 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.435914    4882 scope.go:122] "RemoveContainer" containerID="21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2"
Nov 01 04:20:09.448732 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.448633    4882 scope.go:122] "RemoveContainer" containerID="21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2"
Nov 01 04:20:09.449025 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:20:09.448993    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2\": container with ID starting with 21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2 not found: ID does not exist" containerID="21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2"
Nov 01 04:20:09.449155 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.449035    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2"} err="failed to get container status \"21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2\": rpc error: code = NotFound desc = could not find container \"21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2\": container with ID starting with 21999762606d805f0cd93d2841dbc2a6abb9a225aed448533a94c0a8e399d4d2 not found: ID does not exist"
Nov 01 04:20:09.618606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.618568    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 01 04:20:09.618788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.618622    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 01 04:20:09.673512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.669534    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:20:09.720460 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.720396    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"]
Nov 01 04:20:09.720992 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.720972    4882 cpu_manager.go:414] "containerMap: removing container" podUID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" containerName="istio-proxy"
Nov 01 04:20:09.721093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.720993    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" containerName="istio-proxy"
Nov 01 04:20:09.721093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.721044    4882 cpu_manager.go:414] "containerMap: removing container" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy"
Nov 01 04:20:09.721093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.721054    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy"
Nov 01 04:20:09.721093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.721065    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerName="istio-proxy"
Nov 01 04:20:09.721093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.721073    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerName="istio-proxy"
Nov 01 04:20:09.721093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.721086    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" containerName="istio-proxy"
Nov 01 04:20:09.721420 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.721094    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" containerName="istio-proxy"
Nov 01 04:20:09.721420 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.721280    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" containerName="istio-proxy"
Nov 01 04:20:09.721420 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.721303    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" containerName="istio-proxy"
Nov 01 04:20:09.721420 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.721313    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" containerName="istio-proxy"
Nov 01 04:20:09.721420 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.721324    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" containerName="istio-proxy"
Nov 01 04:20:09.726722 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.726701    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:20:09.734150 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.734130    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"]
Nov 01 04:20:09.742442 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.740050    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2\" (UID: \"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:20:09.742442 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.740095    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cj2h\" (UniqueName: \"kubernetes.io/projected/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-kube-api-access-2cj2h\") pod \"payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2\" (UID: \"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:20:09.742442 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.740958    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-flvv6"]
Nov 01 04:20:09.789451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.784257    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2afc0d\""
Nov 01 04:20:09.789451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.784473    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2afc0d-dockercfg-bwkhw\""
Nov 01 04:20:09.843699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.823416    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"]
Nov 01 04:20:09.879518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.852901    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2\" (UID: \"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:20:09.972515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.958940    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cj2h\" (UniqueName: \"kubernetes.io/projected/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-kube-api-access-2cj2h\") pod \"payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2\" (UID: \"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:20:09.972515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.931257    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"]
Nov 01 04:20:09.972515 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:09.863353    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2\" (UID: \"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:20:10.030212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.022811    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-plhn5"]
Nov 01 04:20:10.030212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.027714    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"]
Nov 01 04:20:10.034829 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.034784    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"]
Nov 01 04:20:10.035007 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.034899    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:20:10.036168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.036143    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cj2h\" (UniqueName: \"kubernetes.io/projected/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-kube-api-access-2cj2h\") pod \"payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2\" (UID: \"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:20:10.036774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.036754    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"]
Nov 01 04:20:10.040983 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.040959    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-f6565"]
Nov 01 04:20:10.042643 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.042623    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:20:10.143423 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.143384    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"]
Nov 01 04:20:10.148571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.148542    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-d9tgt"]
Nov 01 04:20:10.168459 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.166269    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjbp5\" (UniqueName: \"kubernetes.io/projected/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-kube-api-access-xjbp5\") pod \"payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb\" (UID: \"296c6127-1cf0-4538-9cd3-78a41fc4a9f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:20:10.168459 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.166575    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb\" (UID: \"296c6127-1cf0-4538-9cd3-78a41fc4a9f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:20:10.301053 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.300953    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb\" (UID: \"296c6127-1cf0-4538-9cd3-78a41fc4a9f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:20:10.301372 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.301342    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjbp5\" (UniqueName: \"kubernetes.io/projected/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-kube-api-access-xjbp5\") pod \"payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb\" (UID: \"296c6127-1cf0-4538-9cd3-78a41fc4a9f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:20:10.311832 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.302978    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb\" (UID: \"296c6127-1cf0-4538-9cd3-78a41fc4a9f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:20:10.360684 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.360661    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjbp5\" (UniqueName: \"kubernetes.io/projected/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-kube-api-access-xjbp5\") pod \"payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb\" (UID: \"296c6127-1cf0-4538-9cd3-78a41fc4a9f2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:20:10.427325 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.427288    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:20:10.523687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.523627    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"]
Nov 01 04:20:10.618415 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.617579    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 01 04:20:10.618415 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.617645    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 01 04:20:10.851799 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:10.851773    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"]
Nov 01 04:20:10.853583 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:20:10.853551    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296c6127_1cf0_4538_9cd3_78a41fc4a9f2.slice/crio-4477e19cc55bfeb50a235432c78aac625ea25d478f027d3a574fa6ddb63ad736 WatchSource:0}: Error finding container 4477e19cc55bfeb50a235432c78aac625ea25d478f027d3a574fa6ddb63ad736: Status 404 returned error can't find the container with id 4477e19cc55bfeb50a235432c78aac625ea25d478f027d3a574fa6ddb63ad736
Nov 01 04:20:11.356826 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.356787    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb" event={"ID":"296c6127-1cf0-4538-9cd3-78a41fc4a9f2","Type":"ContainerStarted","Data":"f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6"}
Nov 01 04:20:11.356987 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.356863    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb" event={"ID":"296c6127-1cf0-4538-9cd3-78a41fc4a9f2","Type":"ContainerStarted","Data":"4477e19cc55bfeb50a235432c78aac625ea25d478f027d3a574fa6ddb63ad736"}
Nov 01 04:20:11.356987 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.356896    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:20:11.358689 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.358659    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2" event={"ID":"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd","Type":"ContainerStarted","Data":"5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87"}
Nov 01 04:20:11.358800 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.358708    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2" event={"ID":"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd","Type":"ContainerStarted","Data":"2e207bf920a23e83dbb01fffb87774c3499fccb545086ed2a80d8c9e0892e6b2"}
Nov 01 04:20:11.358800 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.358739    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:20:11.597452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.592314    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb" podStartSLOduration=2.592292274 podStartE2EDuration="2.592292274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:20:11.479665283 +0000 UTC m=+2280.041698303" watchObservedRunningTime="2026-11-01 04:20:11.592292274 +0000 UTC m=+2280.154325293"
Nov 01 04:20:11.617262 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.616510    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 01 04:20:11.617262 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.616582    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 01 04:20:11.641458 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.641379    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2" podStartSLOduration=2.641360495 podStartE2EDuration="2.641360495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:20:11.589853669 +0000 UTC m=+2280.151886687" watchObservedRunningTime="2026-11-01 04:20:11.641360495 +0000 UTC m=+2280.203393513"
Nov 01 04:20:11.646139 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.646113    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fbfedcc-725a-4408-883b-62a2c49e655e" path="/var/lib/kubelet/pods/1fbfedcc-725a-4408-883b-62a2c49e655e/volumes"
Nov 01 04:20:11.646660 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.646642    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821ecb15-ab45-430e-8fc7-b7b1d9b069fd" path="/var/lib/kubelet/pods/821ecb15-ab45-430e-8fc7-b7b1d9b069fd/volumes"
Nov 01 04:20:11.647076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.647063    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c0d040-6d16-4890-9595-43e09ef4f0b0" path="/var/lib/kubelet/pods/86c0d040-6d16-4890-9595-43e09ef4f0b0/volumes"
Nov 01 04:20:11.647953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:11.647937    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4921413-f98d-4883-a532-a4e1c45219d7" path="/var/lib/kubelet/pods/a4921413-f98d-4883-a532-a4e1c45219d7/volumes"
Nov 01 04:20:12.593965 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:12.593927    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 01 04:20:12.594402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:12.593984    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 01 04:20:13.046467 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:13.046417    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"]
Nov 01 04:20:13.092306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:13.092279    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cab5f60-n64lt"]
Nov 01 04:20:13.325328 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:13.325252    4882 scope.go:122] "RemoveContainer" containerID="dcddb239b1ef7cc6d1a1a8c77dcb6f89776cf00e0711be1ef033b7c0aae35353"
Nov 01 04:20:13.602782 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:13.602702    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e38030cd-cbfc-4729-b270-e456184ec6d4" path="/var/lib/kubelet/pods/e38030cd-cbfc-4729-b270-e456184ec6d4/volumes"
Nov 01 04:20:13.603261 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:13.603206    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:20:13.604493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:13.604473    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:20:16.432658 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:16.432577    4882 generic.go:356] "Generic (PLEG): container finished" podID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerID="80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa" exitCode=0
Nov 01 04:20:16.433069 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:16.432650    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" event={"ID":"c3109f72-8937-4f69-b7f0-25b2e4870336","Type":"ContainerDied","Data":"80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa"}
Nov 01 04:20:16.433069 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:16.433059    4882 scope.go:122] "RemoveContainer" containerID="80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa"
Nov 01 04:20:16.434708 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:16.434684    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:16.434791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:16.434749    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:16.434791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:16.434780    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:17.449161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:17.449119    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" event={"ID":"c3109f72-8937-4f69-b7f0-25b2e4870336","Type":"ContainerStarted","Data":"248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1"}
Nov 01 04:20:17.888847 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:17.888742    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:20:17.889126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:17.889100    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 01 04:20:17.889212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:17.889155    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 01 04:20:18.889728 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:18.889695    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 01 04:20:18.890184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:18.889749    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 01 04:20:19.889520 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:19.889490    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 01 04:20:19.889773 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:19.889751    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 01 04:20:20.889598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:20.889512    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 01 04:20:20.889598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:20.889579    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 01 04:20:21.888937 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:21.888898    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 01 04:20:21.889413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:21.888966    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 01 04:20:22.404059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:22.404032    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:20:22.413667 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:22.413646    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:20:22.889322 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:22.889287    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 01 04:20:22.889765 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:22.889353    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 01 04:20:23.045514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:23.045473    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"]
Nov 01 04:20:23.048619 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:23.048595    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732324a5-5bcnf"]
Nov 01 04:20:23.600479 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:23.600449    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093dd5d0-8f3a-4d24-9800-af71c61a4b3b" path="/var/lib/kubelet/pods/093dd5d0-8f3a-4d24-9800-af71c61a4b3b/volumes"
Nov 01 04:20:23.893962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:23.893882    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:20:23.894416 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:23.894393    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:20:23.895027 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:23.895007    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:20:26.126462 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.126406    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"]
Nov 01 04:20:26.175451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.174692    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.191221 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.191186    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"]
Nov 01 04:20:26.234413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.234384    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-862ad1-openshift-default-dockercfg-psn88\""
Nov 01 04:20:26.305740 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.305709    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-socket\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.305926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.305752    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-credential-socket\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.305926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.305864    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istiod-ca-cert\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.306012 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.305923    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-token\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.306012 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.305998    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-podinfo\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.306078 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.306051    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdz8f\" (UniqueName: \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-kube-api-access-pdz8f\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.306135 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.306121    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-certs\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.306177 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.306141    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-envoy\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.306177 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.306165    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-data\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.407476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.407182    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-podinfo\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.407476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.407271    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdz8f\" (UniqueName: \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-kube-api-access-pdz8f\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.407476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.407332    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-certs\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.407476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.407356    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-envoy\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.407476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.407394    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-data\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.407907 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.407500    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-socket\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.407907 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.407542    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-credential-socket\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.407907 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.407610    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istiod-ca-cert\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.407907 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.407686    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-token\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.409184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.408825    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-data\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.409660 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.409343    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-certs\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.411892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.411149    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-credential-socket\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.411892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.411652    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-socket\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.411892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.411811    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-envoy\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.412086 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.411931    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istiod-ca-cert\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.412216 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.412135    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-podinfo\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.464971 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.464935    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-token\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.465225 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.465203    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdz8f\" (UniqueName: \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-kube-api-access-pdz8f\") pod \"e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") " pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.508796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.508758    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:26.934764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.934740    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"]
Nov 01 04:20:26.936133 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:20:26.936104    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ce7003_30a0_4d13_a9be_cf28b38a1a7d.slice/crio-6f6cfd3453f0a3b37c97c700c66ae6c787cb3edbdf284b2f103dcd6811271e8f WatchSource:0}: Error finding container 6f6cfd3453f0a3b37c97c700c66ae6c787cb3edbdf284b2f103dcd6811271e8f: Status 404 returned error can't find the container with id 6f6cfd3453f0a3b37c97c700c66ae6c787cb3edbdf284b2f103dcd6811271e8f
Nov 01 04:20:26.938886 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.938853    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:26.938958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.938920    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:26.938958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:26.938947    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:20:27.545963 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:27.545923    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" event={"ID":"81ce7003-30a0-4d13-a9be-cf28b38a1a7d","Type":"ContainerStarted","Data":"0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595"}
Nov 01 04:20:27.546363 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:27.546016    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" event={"ID":"81ce7003-30a0-4d13-a9be-cf28b38a1a7d","Type":"ContainerStarted","Data":"6f6cfd3453f0a3b37c97c700c66ae6c787cb3edbdf284b2f103dcd6811271e8f"}
Nov 01 04:20:27.659336 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:27.659276    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" podStartSLOduration=1.659261275 podStartE2EDuration="1.659261275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:20:27.658027628 +0000 UTC m=+2296.220060645" watchObservedRunningTime="2026-11-01 04:20:27.659261275 +0000 UTC m=+2296.221294269"
Nov 01 04:20:28.509631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:28.509598    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:28.515748 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:28.515722    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:28.552529 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:28.552490    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:28.553458 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:20:28.553440    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:20:28.723205 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:20:28.723167    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:20:58.723160 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:20:58.723119    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:21:11.447403 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.447370    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"]
Nov 01 04:21:11.451442 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.451409    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:21:11.547718 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.547682    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a7b5f0-dockercfg-hsfj8\""
Nov 01 04:21:11.547904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.547771    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a7b5f0\""
Nov 01 04:21:11.550970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.550945    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"]
Nov 01 04:21:11.591087 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.591058    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9dw\" (UniqueName: \"kubernetes.io/projected/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-kube-api-access-9b9dw\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57\" (UID: \"764e0ac7-3fa3-4fb7-a7c6-4592b8304930\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:21:11.591241 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.591154    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57\" (UID: \"764e0ac7-3fa3-4fb7-a7c6-4592b8304930\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:21:11.692453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.692408    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57\" (UID: \"764e0ac7-3fa3-4fb7-a7c6-4592b8304930\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:21:11.692622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.692572    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9dw\" (UniqueName: \"kubernetes.io/projected/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-kube-api-access-9b9dw\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57\" (UID: \"764e0ac7-3fa3-4fb7-a7c6-4592b8304930\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:21:11.693301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.693280    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57\" (UID: \"764e0ac7-3fa3-4fb7-a7c6-4592b8304930\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:21:11.750194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.750122    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9dw\" (UniqueName: \"kubernetes.io/projected/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-kube-api-access-9b9dw\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57\" (UID: \"764e0ac7-3fa3-4fb7-a7c6-4592b8304930\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:21:11.765098 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:11.765075    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:21:12.140661 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.140629    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"]
Nov 01 04:21:12.142381 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:21:12.142357    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764e0ac7_3fa3_4fb7_a7c6_4592b8304930.slice/crio-4baab1503f05e34d5b39b7f277f1f8b27ebb0d64a679db069e5775e4846767e1 WatchSource:0}: Error finding container 4baab1503f05e34d5b39b7f277f1f8b27ebb0d64a679db069e5775e4846767e1: Status 404 returned error can't find the container with id 4baab1503f05e34d5b39b7f277f1f8b27ebb0d64a679db069e5775e4846767e1
Nov 01 04:21:12.212141 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.212117    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"]
Nov 01 04:21:12.216137 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.216115    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:21:12.270095 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.270066    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"]
Nov 01 04:21:12.301390 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.301358    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6\" (UID: \"7808dd74-e5e3-45c8-a8da-eb47fe119cfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:21:12.301390 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.301395    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgnv\" (UniqueName: \"kubernetes.io/projected/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-kube-api-access-ncgnv\") pod \"payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6\" (UID: \"7808dd74-e5e3-45c8-a8da-eb47fe119cfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:21:12.402362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.402270    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6\" (UID: \"7808dd74-e5e3-45c8-a8da-eb47fe119cfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:21:12.402362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.402317    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgnv\" (UniqueName: \"kubernetes.io/projected/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-kube-api-access-ncgnv\") pod \"payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6\" (UID: \"7808dd74-e5e3-45c8-a8da-eb47fe119cfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:21:12.403447 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.403410    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6\" (UID: \"7808dd74-e5e3-45c8-a8da-eb47fe119cfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:21:12.459346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.459319    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgnv\" (UniqueName: \"kubernetes.io/projected/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-kube-api-access-ncgnv\") pod \"payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6\" (UID: \"7808dd74-e5e3-45c8-a8da-eb47fe119cfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:21:12.529169 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.529136    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:21:12.855623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.855587    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57" event={"ID":"764e0ac7-3fa3-4fb7-a7c6-4592b8304930","Type":"ContainerStarted","Data":"ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa"}
Nov 01 04:21:12.855805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.855639    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57" event={"ID":"764e0ac7-3fa3-4fb7-a7c6-4592b8304930","Type":"ContainerStarted","Data":"4baab1503f05e34d5b39b7f277f1f8b27ebb0d64a679db069e5775e4846767e1"}
Nov 01 04:21:12.855805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.855676    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:21:12.902608 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.902579    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"]
Nov 01 04:21:12.904137 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:21:12.904106    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7808dd74_e5e3_45c8_a8da_eb47fe119cfe.slice/crio-aa34a11d185d8349f6dcaa757c6697d2a1ad3f283f7f0eef8c6f96f7c1a8660f WatchSource:0}: Error finding container aa34a11d185d8349f6dcaa757c6697d2a1ad3f283f7f0eef8c6f96f7c1a8660f: Status 404 returned error can't find the container with id aa34a11d185d8349f6dcaa757c6697d2a1ad3f283f7f0eef8c6f96f7c1a8660f
Nov 01 04:21:12.967184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:12.967138    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57" podStartSLOduration=1.9671208409999998 podStartE2EDuration="1.967120841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:21:12.963698888 +0000 UTC m=+2341.525731907" watchObservedRunningTime="2026-11-01 04:21:12.967120841 +0000 UTC m=+2341.529153859"
Nov 01 04:21:13.375281 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:13.375246    4882 scope.go:122] "RemoveContainer" containerID="97e4e08a0e7eb3633f4b50ef0c24509001eb163c1eb9448ef0b2be1314646662"
Nov 01 04:21:13.866997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:13.866947    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6" event={"ID":"7808dd74-e5e3-45c8-a8da-eb47fe119cfe","Type":"ContainerStarted","Data":"ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049"}
Nov 01 04:21:13.867409 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:13.867020    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6" event={"ID":"7808dd74-e5e3-45c8-a8da-eb47fe119cfe","Type":"ContainerStarted","Data":"aa34a11d185d8349f6dcaa757c6697d2a1ad3f283f7f0eef8c6f96f7c1a8660f"}
Nov 01 04:21:13.867409 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:13.867111    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:21:13.979342 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:13.979275    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6" podStartSLOduration=1.979256027 podStartE2EDuration="1.979256027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:21:13.975324907 +0000 UTC m=+2342.537357925" watchObservedRunningTime="2026-11-01 04:21:13.979256027 +0000 UTC m=+2342.541289046"
Nov 01 04:21:23.886700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:23.886647    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:21:24.886860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:24.886823    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:21:28.390448 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.390398    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"]
Nov 01 04:21:28.397927 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.397896    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:28.433089 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.433055    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fmt\" (UniqueName: \"kubernetes.io/projected/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-kube-api-access-k2fmt\") pod \"maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:28.433280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.433250    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:28.433347 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.433333    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:28.449865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.449820    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"]
Nov 01 04:21:28.490787 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.490753    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:21:28.490958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.490792    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:21:28.535453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.535382    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fmt\" (UniqueName: \"kubernetes.io/projected/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-kube-api-access-k2fmt\") pod \"maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:28.535655 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.535553    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:28.535655 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.535611    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:28.536339 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.536312    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:28.538099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.538067    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:28.595178 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.595141    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fmt\" (UniqueName: \"kubernetes.io/projected/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-kube-api-access-k2fmt\") pod \"maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:28.712287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:28.712248    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:28.725409 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:21:28.725372    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:21:29.094085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:29.093158    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"]
Nov 01 04:21:29.094693 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:21:29.094649    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf526fe0f_78ff_45d9_9820_be2cbb1fe10e.slice/crio-82e81169a7f6949a1bf76bb17751797eb203df1a673c086a26c1e6f099c2a6e5 WatchSource:0}: Error finding container 82e81169a7f6949a1bf76bb17751797eb203df1a673c086a26c1e6f099c2a6e5: Status 404 returned error can't find the container with id 82e81169a7f6949a1bf76bb17751797eb203df1a673c086a26c1e6f099c2a6e5
Nov 01 04:21:30.022960 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:30.022920    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx" event={"ID":"f526fe0f-78ff-45d9-9820-be2cbb1fe10e","Type":"ContainerStarted","Data":"5cbe3867ccbc212121c092952db6f12bb49fd52e75c355844d8ba8e71fc3e305"}
Nov 01 04:21:30.023504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:30.023027    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx" event={"ID":"f526fe0f-78ff-45d9-9820-be2cbb1fe10e","Type":"ContainerStarted","Data":"82e81169a7f6949a1bf76bb17751797eb203df1a673c086a26c1e6f099c2a6e5"}
Nov 01 04:21:30.133723 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:30.133660    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx" podStartSLOduration=2.133641023 podStartE2EDuration="2.133641023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:21:30.12974552 +0000 UTC m=+2358.691778539" watchObservedRunningTime="2026-11-01 04:21:30.133641023 +0000 UTC m=+2358.695674042"
Nov 01 04:21:31.031175 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:31.031142    4882 generic.go:356] "Generic (PLEG): container finished" podID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerID="28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181" exitCode=0
Nov 01 04:21:31.031682 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:31.031198    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" event={"ID":"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b","Type":"ContainerDied","Data":"28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181"}
Nov 01 04:21:31.031755 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:31.031704    4882 scope.go:122] "RemoveContainer" containerID="28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181"
Nov 01 04:21:31.032913 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:31.032888    4882 generic.go:356] "Generic (PLEG): container finished" podID="f526fe0f-78ff-45d9-9820-be2cbb1fe10e" containerID="5cbe3867ccbc212121c092952db6f12bb49fd52e75c355844d8ba8e71fc3e305" exitCode=0
Nov 01 04:21:31.033015 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:31.032918    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx" event={"ID":"f526fe0f-78ff-45d9-9820-be2cbb1fe10e","Type":"ContainerDied","Data":"5cbe3867ccbc212121c092952db6f12bb49fd52e75c355844d8ba8e71fc3e305"}
Nov 01 04:21:31.033407 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:31.033382    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:21:31.033470 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:31.033457    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:21:31.033513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:31.033490    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:21:31.385633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:31.385543    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6_7808dd74-e5e3-45c8-a8da-eb47fe119cfe/payload-processing/0.log"
Nov 01 04:21:32.040945 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.040908    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" event={"ID":"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b","Type":"ContainerStarted","Data":"cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719"}
Nov 01 04:21:32.259085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.259054    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:32.396477 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.396353    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-openshift-ca\") pod \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") "
Nov 01 04:21:32.396477 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.396471    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-kube-api-access-k2fmt\" (UniqueName: \"kubernetes.io/projected/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-kube-api-access-k2fmt\") pod \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") "
Nov 01 04:21:32.396734 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.396699    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-xks-ca\") pod \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\" (UID: \"f526fe0f-78ff-45d9-9820-be2cbb1fe10e\") "
Nov 01 04:21:32.397944 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.397914    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-openshift-ca" pod "f526fe0f-78ff-45d9-9820-be2cbb1fe10e" (UID: "f526fe0f-78ff-45d9-9820-be2cbb1fe10e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:21:32.401084 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.400793    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-xks-ca" pod "f526fe0f-78ff-45d9-9820-be2cbb1fe10e" (UID: "f526fe0f-78ff-45d9-9820-be2cbb1fe10e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:21:32.401084 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.400811    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-kube-api-access-k2fmt" pod "f526fe0f-78ff-45d9-9820-be2cbb1fe10e" (UID: "f526fe0f-78ff-45d9-9820-be2cbb1fe10e"). InnerVolumeSpecName "kube-api-access-k2fmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:21:32.498490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.498449    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:21:32.498490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.498483    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:21:32.498490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.498496    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2fmt\" (UniqueName: \"kubernetes.io/projected/f526fe0f-78ff-45d9-9820-be2cbb1fe10e-kube-api-access-k2fmt\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:21:32.675513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.669876    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"]
Nov 01 04:21:32.675513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.670801    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f526fe0f-78ff-45d9-9820-be2cbb1fe10e" containerName="revoke-keys"
Nov 01 04:21:32.675513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.670823    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f526fe0f-78ff-45d9-9820-be2cbb1fe10e" containerName="revoke-keys"
Nov 01 04:21:32.675513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.671018    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f526fe0f-78ff-45d9-9820-be2cbb1fe10e" containerName="revoke-keys"
Nov 01 04:21:32.690461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.690310    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:32.721489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.721462    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"]
Nov 01 04:21:32.801479 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.801444    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kv92\" (UniqueName: \"kubernetes.io/projected/1c3a833a-14a3-4351-88a6-e13c6d803fe5-kube-api-access-6kv92\") pod \"maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:32.801660 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.801606    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:32.801718 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.801676    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:32.839175 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.839142    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:21:32.840414 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.840388    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 01 04:21:32.840566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.840456    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 01 04:21:32.903384 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.903347    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:32.903625 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.903414    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:32.903625 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.903555    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kv92\" (UniqueName: \"kubernetes.io/projected/1c3a833a-14a3-4351-88a6-e13c6d803fe5-kube-api-access-6kv92\") pod \"maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:32.904145 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.904117    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:32.905864 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.905844    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:32.984195 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:32.984157    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kv92\" (UniqueName: \"kubernetes.io/projected/1c3a833a-14a3-4351-88a6-e13c6d803fe5-kube-api-access-6kv92\") pod \"maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:33.006745 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:33.006717    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:33.049856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:33.049643    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx" event={"ID":"f526fe0f-78ff-45d9-9820-be2cbb1fe10e","Type":"ContainerDied","Data":"82e81169a7f6949a1bf76bb17751797eb203df1a673c086a26c1e6f099c2a6e5"}
Nov 01 04:21:33.049856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:33.049694    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"
Nov 01 04:21:33.050373 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:33.049700    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82e81169a7f6949a1bf76bb17751797eb203df1a673c086a26c1e6f099c2a6e5"
Nov 01 04:21:33.402820 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:33.402796    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"]
Nov 01 04:21:33.404785 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:21:33.404757    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3a833a_14a3_4351_88a6_e13c6d803fe5.slice/crio-2bcaa44fc99cdc2a07de10fa9f71577d1c1956609db41e02ef8f3dc22cd04ce6 WatchSource:0}: Error finding container 2bcaa44fc99cdc2a07de10fa9f71577d1c1956609db41e02ef8f3dc22cd04ce6: Status 404 returned error can't find the container with id 2bcaa44fc99cdc2a07de10fa9f71577d1c1956609db41e02ef8f3dc22cd04ce6
Nov 01 04:21:33.780572 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:33.780539    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:21:33.842902 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:33.842860    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 01 04:21:33.843182 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:33.843147    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 01 04:21:34.057175 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:34.057074    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn" event={"ID":"1c3a833a-14a3-4351-88a6-e13c6d803fe5","Type":"ContainerStarted","Data":"42ea6f18004fe3ac8c4c3e0c0b08f2d8ec2a444d77a7d4b44063d71380bb3064"}
Nov 01 04:21:34.057175 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:34.057135    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn" event={"ID":"1c3a833a-14a3-4351-88a6-e13c6d803fe5","Type":"ContainerStarted","Data":"2bcaa44fc99cdc2a07de10fa9f71577d1c1956609db41e02ef8f3dc22cd04ce6"}
Nov 01 04:21:34.175453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:34.170845    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn" podStartSLOduration=2.170826755 podStartE2EDuration="2.170826755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:21:34.168673343 +0000 UTC m=+2362.730706364" watchObservedRunningTime="2026-11-01 04:21:34.170826755 +0000 UTC m=+2362.732859773"
Nov 01 04:21:34.839544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:34.839423    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 01 04:21:34.839544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:34.839510    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 01 04:21:35.088536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:35.088502    4882 generic.go:356] "Generic (PLEG): container finished" podID="1c3a833a-14a3-4351-88a6-e13c6d803fe5" containerID="42ea6f18004fe3ac8c4c3e0c0b08f2d8ec2a444d77a7d4b44063d71380bb3064" exitCode=0
Nov 01 04:21:35.089000 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:35.088604    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn" event={"ID":"1c3a833a-14a3-4351-88a6-e13c6d803fe5","Type":"ContainerDied","Data":"42ea6f18004fe3ac8c4c3e0c0b08f2d8ec2a444d77a7d4b44063d71380bb3064"}
Nov 01 04:21:35.839037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:35.838999    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 01 04:21:35.839208 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:35.839083    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 01 04:21:36.292038 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.292004    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:36.346069 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.345977    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3a833a-14a3-4351-88a6-e13c6d803fe5-kube-api-access-6kv92\" (UniqueName: \"kubernetes.io/projected/1c3a833a-14a3-4351-88a6-e13c6d803fe5-kube-api-access-6kv92\") pod \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") "
Nov 01 04:21:36.346247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.346178    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-openshift-ca\") pod \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") "
Nov 01 04:21:36.346319 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.346252    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-xks-ca\") pod \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\" (UID: \"1c3a833a-14a3-4351-88a6-e13c6d803fe5\") "
Nov 01 04:21:36.346598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.346564    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-openshift-ca" pod "1c3a833a-14a3-4351-88a6-e13c6d803fe5" (UID: "1c3a833a-14a3-4351-88a6-e13c6d803fe5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:21:36.346801 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.346786    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:21:36.348193 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.348161    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3a833a-14a3-4351-88a6-e13c6d803fe5-kube-api-access-6kv92" pod "1c3a833a-14a3-4351-88a6-e13c6d803fe5" (UID: "1c3a833a-14a3-4351-88a6-e13c6d803fe5"). InnerVolumeSpecName "kube-api-access-6kv92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:21:36.348290 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.348204    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-xks-ca" pod "1c3a833a-14a3-4351-88a6-e13c6d803fe5" (UID: "1c3a833a-14a3-4351-88a6-e13c6d803fe5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:21:36.448600 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.448567    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3a833a-14a3-4351-88a6-e13c6d803fe5-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:21:36.448600 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.448593    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kv92\" (UniqueName: \"kubernetes.io/projected/1c3a833a-14a3-4351-88a6-e13c6d803fe5-kube-api-access-6kv92\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:21:36.839103 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.839068    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Nov 01 04:21:36.839280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:36.839124    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Nov 01 04:21:37.103111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:37.102999    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"
Nov 01 04:21:37.103111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:37.103032    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn" event={"ID":"1c3a833a-14a3-4351-88a6-e13c6d803fe5","Type":"ContainerDied","Data":"2bcaa44fc99cdc2a07de10fa9f71577d1c1956609db41e02ef8f3dc22cd04ce6"}
Nov 01 04:21:37.103111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:37.103109    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bcaa44fc99cdc2a07de10fa9f71577d1c1956609db41e02ef8f3dc22cd04ce6"
Nov 01 04:21:37.844245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:37.844216    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:21:37.845229 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:21:37.845208    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:21:58.723258 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:21:58.723178    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:22:04.706089 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:04.705122    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": read tcp 10.132.0.2:45632->10.132.0.176:15021: read: connection reset by peer" start-of-body=
Nov 01 04:22:04.706089 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:04.705180    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": read tcp 10.132.0.2:45632->10.132.0.176:15021: read: connection reset by peer"
Nov 01 04:22:05.352599 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:05.352562    4882 generic.go:356] "Generic (PLEG): container finished" podID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerID="3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee" exitCode=0
Nov 01 04:22:05.352785 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:05.352636    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" event={"ID":"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b","Type":"ContainerDied","Data":"3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee"}
Nov 01 04:22:05.353042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:05.353025    4882 scope.go:122] "RemoveContainer" containerID="3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee"
Nov 01 04:22:05.354570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:05.354544    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:22:05.354629 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:05.354612    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:22:05.354668 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:05.354645    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:22:06.361378 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:06.361346    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" event={"ID":"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b","Type":"ContainerStarted","Data":"3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255"}
Nov 01 04:22:07.350835 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:07.350799    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:22:07.351069 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:07.351046    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 01 04:22:07.351153 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:07.351100    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 01 04:22:08.351596 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:08.351559    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 01 04:22:08.352033 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:08.351646    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 01 04:22:09.351239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:09.351196    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 01 04:22:09.351455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:09.351255    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 01 04:22:10.350880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:10.350843    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 01 04:22:10.351280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:10.350903    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 01 04:22:11.351229 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:11.351194    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 01 04:22:11.351647 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:11.351277    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 01 04:22:12.355992 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:12.355956    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:22:12.356494 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:12.356454    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:22:12.357165 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:12.357145    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:22:13.035099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:13.035069    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:22:13.038559 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:13.038535    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:22:13.406899 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:13.406820    4882 scope.go:122] "RemoveContainer" containerID="359b78135246edae54f29038f8d8bd0f16c74b6f57baefef71d4fc4acbbb692b"
Nov 01 04:22:17.519008 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:17.518968    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"]
Nov 01 04:22:17.522089 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:17.522060    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-131718-ghmsn"]
Nov 01 04:22:17.600772 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:17.600735    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3a833a-14a3-4351-88a6-e13c6d803fe5" path="/var/lib/kubelet/pods/1c3a833a-14a3-4351-88a6-e13c6d803fe5/volumes"
Nov 01 04:22:24.135028 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:24.134993    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"]
Nov 01 04:22:24.135526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:24.135246    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy" containerID="cri-o://acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1" gracePeriod=30
Nov 01 04:22:25.021523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.021486    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"]
Nov 01 04:22:25.022218 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.022197    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1c3a833a-14a3-4351-88a6-e13c6d803fe5" containerName="revoke-keys"
Nov 01 04:22:25.022218 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.022216    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3a833a-14a3-4351-88a6-e13c6d803fe5" containerName="revoke-keys"
Nov 01 04:22:25.022423 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.022307    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3a833a-14a3-4351-88a6-e13c6d803fe5" containerName="revoke-keys"
Nov 01 04:22:25.027261 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.027238    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:25.076785 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.076745    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:22:25.080472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.080415    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"]
Nov 01 04:22:25.122712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.122680    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:22:25.155008 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.154975    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:25.155375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.155109    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nxgz\" (UniqueName: \"kubernetes.io/projected/51b0ae48-6f64-4968-9931-e636eb4fb1d1-kube-api-access-7nxgz\") pod \"maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:25.155375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.155174    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:25.256458 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.256391    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:25.256655 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.256533    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nxgz\" (UniqueName: \"kubernetes.io/projected/51b0ae48-6f64-4968-9931-e636eb4fb1d1-kube-api-access-7nxgz\") pod \"maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:25.256655 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.256584    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:25.257136 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.257112    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:25.258884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.258862    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:25.313243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.313176    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nxgz\" (UniqueName: \"kubernetes.io/projected/51b0ae48-6f64-4968-9931-e636eb4fb1d1-kube-api-access-7nxgz\") pod \"maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:25.340786 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.340752    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:25.715550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:25.715518    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"]
Nov 01 04:22:25.717090 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:22:25.717061    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b0ae48_6f64_4968_9931_e636eb4fb1d1.slice/crio-290fd5fec997155b1cb1c77a295532d207e1f44beeef5f2cdff513f11fbab1a6 WatchSource:0}: Error finding container 290fd5fec997155b1cb1c77a295532d207e1f44beeef5f2cdff513f11fbab1a6: Status 404 returned error can't find the container with id 290fd5fec997155b1cb1c77a295532d207e1f44beeef5f2cdff513f11fbab1a6
Nov 01 04:22:26.527661 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:26.527625    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8" event={"ID":"51b0ae48-6f64-4968-9931-e636eb4fb1d1","Type":"ContainerStarted","Data":"ba24c9bf7bd0e7a052b1416c36b206cd46235959c5dff795ddacb971b4358a9c"}
Nov 01 04:22:26.528094 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:26.527685    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8" event={"ID":"51b0ae48-6f64-4968-9931-e636eb4fb1d1","Type":"ContainerStarted","Data":"290fd5fec997155b1cb1c77a295532d207e1f44beeef5f2cdff513f11fbab1a6"}
Nov 01 04:22:26.639304 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:26.636399    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8" podStartSLOduration=2.636381733 podStartE2EDuration="2.636381733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:22:26.635513638 +0000 UTC m=+2415.197546656" watchObservedRunningTime="2026-11-01 04:22:26.636381733 +0000 UTC m=+2415.198414755"
Nov 01 04:22:27.410510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.410472    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"]
Nov 01 04:22:27.410815 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.410757    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw" podUID="9251b815-0f89-4f7c-bba6-046d34815b35" containerName="payload-processing" containerID="cri-o://60128db4946542f534e6c5b80028c7d9ddd415141433c913f912510435961ca7" gracePeriod=30
Nov 01 04:22:27.539594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.539519    4882 patch_prober.go:29] interesting pod/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:22:27.540013 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.539596    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:22:27.547060 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.547031    4882 generic.go:356] "Generic (PLEG): container finished" podID="9251b815-0f89-4f7c-bba6-046d34815b35" containerID="60128db4946542f534e6c5b80028c7d9ddd415141433c913f912510435961ca7" exitCode=0
Nov 01 04:22:27.547193 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.547098    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw" event={"ID":"9251b815-0f89-4f7c-bba6-046d34815b35","Type":"ContainerDied","Data":"60128db4946542f534e6c5b80028c7d9ddd415141433c913f912510435961ca7"}
Nov 01 04:22:27.548987 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.548966    4882 generic.go:356] "Generic (PLEG): container finished" podID="51b0ae48-6f64-4968-9931-e636eb4fb1d1" containerID="ba24c9bf7bd0e7a052b1416c36b206cd46235959c5dff795ddacb971b4358a9c" exitCode=0
Nov 01 04:22:27.549082 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.549015    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8" event={"ID":"51b0ae48-6f64-4968-9931-e636eb4fb1d1","Type":"ContainerDied","Data":"ba24c9bf7bd0e7a052b1416c36b206cd46235959c5dff795ddacb971b4358a9c"}
Nov 01 04:22:27.726934 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.726904    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:22:27.799487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.799346    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9251b815-0f89-4f7c-bba6-046d34815b35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9251b815-0f89-4f7c-bba6-046d34815b35-plugins-config-volume\") pod \"9251b815-0f89-4f7c-bba6-046d34815b35\" (UID: \"9251b815-0f89-4f7c-bba6-046d34815b35\") "
Nov 01 04:22:27.799696 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.799494    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9251b815-0f89-4f7c-bba6-046d34815b35-kube-api-access-4dw6b\" (UniqueName: \"kubernetes.io/projected/9251b815-0f89-4f7c-bba6-046d34815b35-kube-api-access-4dw6b\") pod \"9251b815-0f89-4f7c-bba6-046d34815b35\" (UID: \"9251b815-0f89-4f7c-bba6-046d34815b35\") "
Nov 01 04:22:27.801216 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.801183    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9251b815-0f89-4f7c-bba6-046d34815b35-plugins-config-volume" pod "9251b815-0f89-4f7c-bba6-046d34815b35" (UID: "9251b815-0f89-4f7c-bba6-046d34815b35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:22:27.804352 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.804324    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9251b815-0f89-4f7c-bba6-046d34815b35-kube-api-access-4dw6b" pod "9251b815-0f89-4f7c-bba6-046d34815b35" (UID: "9251b815-0f89-4f7c-bba6-046d34815b35"). InnerVolumeSpecName "kube-api-access-4dw6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:22:27.902047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.902010    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dw6b\" (UniqueName: \"kubernetes.io/projected/9251b815-0f89-4f7c-bba6-046d34815b35-kube-api-access-4dw6b\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:27.902047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:27.902043    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9251b815-0f89-4f7c-bba6-046d34815b35-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:28.557087 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.557055    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"
Nov 01 04:22:28.557087 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.557073    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw" event={"ID":"9251b815-0f89-4f7c-bba6-046d34815b35","Type":"ContainerDied","Data":"118196d58da56b5f2ca031e447db3767b0c4422227c1b1dcd103d94904b15c69"}
Nov 01 04:22:28.557589 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.557150    4882 scope.go:122] "RemoveContainer" containerID="60128db4946542f534e6c5b80028c7d9ddd415141433c913f912510435961ca7"
Nov 01 04:22:28.767746 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:22:28.767709    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:22:28.772148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.772119    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"]
Nov 01 04:22:28.815779 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.815714    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:28.818365 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.818343    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-601a4b-5856bb8598-866dw"]
Nov 01 04:22:28.844855 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.844827    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-openshift-ca\") pod \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") "
Nov 01 04:22:28.845024 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.844953    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b0ae48-6f64-4968-9931-e636eb4fb1d1-kube-api-access-7nxgz\" (UniqueName: \"kubernetes.io/projected/51b0ae48-6f64-4968-9931-e636eb4fb1d1-kube-api-access-7nxgz\") pod \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") "
Nov 01 04:22:28.845024 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.845000    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-xks-ca\") pod \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\" (UID: \"51b0ae48-6f64-4968-9931-e636eb4fb1d1\") "
Nov 01 04:22:28.845201 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.845178    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-openshift-ca" pod "51b0ae48-6f64-4968-9931-e636eb4fb1d1" (UID: "51b0ae48-6f64-4968-9931-e636eb4fb1d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:22:28.845577 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.845554    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:28.847153 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.847123    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-xks-ca" pod "51b0ae48-6f64-4968-9931-e636eb4fb1d1" (UID: "51b0ae48-6f64-4968-9931-e636eb4fb1d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:22:28.847251 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.847227    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b0ae48-6f64-4968-9931-e636eb4fb1d1-kube-api-access-7nxgz" pod "51b0ae48-6f64-4968-9931-e636eb4fb1d1" (UID: "51b0ae48-6f64-4968-9931-e636eb4fb1d1"). InnerVolumeSpecName "kube-api-access-7nxgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:22:28.947005 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.946966    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nxgz\" (UniqueName: \"kubernetes.io/projected/51b0ae48-6f64-4968-9931-e636eb4fb1d1-kube-api-access-7nxgz\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:28.947005 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:28.947007    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b0ae48-6f64-4968-9931-e636eb4fb1d1-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:29.355345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.355308    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"]
Nov 01 04:22:29.358547 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.358517    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-601a4b-6hblx"]
Nov 01 04:22:29.387698 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.387668    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:22:29.456752 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.456716    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-kube-api-access-qhtms\" (UniqueName: \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-kube-api-access-qhtms\") pod \"2c4abcc8-b995-4851-9a00-5514e866ad89\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") "
Nov 01 04:22:29.456912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.456778    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-envoy\") pod \"2c4abcc8-b995-4851-9a00-5514e866ad89\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") "
Nov 01 04:22:29.456912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.456801    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-podinfo\") pod \"2c4abcc8-b995-4851-9a00-5514e866ad89\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") "
Nov 01 04:22:29.456912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.456819    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-token\" (UniqueName: \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-token\") pod \"2c4abcc8-b995-4851-9a00-5514e866ad89\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") "
Nov 01 04:22:29.456912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.456842    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-socket\") pod \"2c4abcc8-b995-4851-9a00-5514e866ad89\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") "
Nov 01 04:22:29.456912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.456899    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-certs\") pod \"2c4abcc8-b995-4851-9a00-5514e866ad89\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") "
Nov 01 04:22:29.457186 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.456974    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-data\") pod \"2c4abcc8-b995-4851-9a00-5514e866ad89\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") "
Nov 01 04:22:29.457186 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.457030    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4abcc8-b995-4851-9a00-5514e866ad89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4abcc8-b995-4851-9a00-5514e866ad89-istiod-ca-cert\") pod \"2c4abcc8-b995-4851-9a00-5514e866ad89\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") "
Nov 01 04:22:29.457186 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.457073    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-credential-socket\") pod \"2c4abcc8-b995-4851-9a00-5514e866ad89\" (UID: \"2c4abcc8-b995-4851-9a00-5514e866ad89\") "
Nov 01 04:22:29.457346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.457235    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-certs" pod "2c4abcc8-b995-4851-9a00-5514e866ad89" (UID: "2c4abcc8-b995-4851-9a00-5514e866ad89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:22:29.457424 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.457384    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-data" pod "2c4abcc8-b995-4851-9a00-5514e866ad89" (UID: "2c4abcc8-b995-4851-9a00-5514e866ad89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:22:29.457566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.457483    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-credential-socket" pod "2c4abcc8-b995-4851-9a00-5514e866ad89" (UID: "2c4abcc8-b995-4851-9a00-5514e866ad89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:22:29.457566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.457495    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-socket" pod "2c4abcc8-b995-4851-9a00-5514e866ad89" (UID: "2c4abcc8-b995-4851-9a00-5514e866ad89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:22:29.457765 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.457744    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:29.457808 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.457774    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:29.457808 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.457774    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4abcc8-b995-4851-9a00-5514e866ad89-istiod-ca-cert" pod "2c4abcc8-b995-4851-9a00-5514e866ad89" (UID: "2c4abcc8-b995-4851-9a00-5514e866ad89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:22:29.457808 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.457790    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:29.457920 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.457804    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:29.459130 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.459111    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-envoy" pod "2c4abcc8-b995-4851-9a00-5514e866ad89" (UID: "2c4abcc8-b995-4851-9a00-5514e866ad89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:22:29.459198 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.459145    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-token" pod "2c4abcc8-b995-4851-9a00-5514e866ad89" (UID: "2c4abcc8-b995-4851-9a00-5514e866ad89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:22:29.459466 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.459446    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-podinfo" pod "2c4abcc8-b995-4851-9a00-5514e866ad89" (UID: "2c4abcc8-b995-4851-9a00-5514e866ad89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:22:29.459508 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.459465    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-kube-api-access-qhtms" pod "2c4abcc8-b995-4851-9a00-5514e866ad89" (UID: "2c4abcc8-b995-4851-9a00-5514e866ad89"). InnerVolumeSpecName "kube-api-access-qhtms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:22:29.558703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.558661    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:29.558703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.558699    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:29.558703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.558708    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:29.559167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.558721    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4abcc8-b995-4851-9a00-5514e866ad89-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:29.559167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.558735    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhtms\" (UniqueName: \"kubernetes.io/projected/2c4abcc8-b995-4851-9a00-5514e866ad89-kube-api-access-qhtms\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:29.566141 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.566106    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8" event={"ID":"51b0ae48-6f64-4968-9931-e636eb4fb1d1","Type":"ContainerDied","Data":"290fd5fec997155b1cb1c77a295532d207e1f44beeef5f2cdff513f11fbab1a6"}
Nov 01 04:22:29.566315 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.566168    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="290fd5fec997155b1cb1c77a295532d207e1f44beeef5f2cdff513f11fbab1a6"
Nov 01 04:22:29.566315 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.566118    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"
Nov 01 04:22:29.568024 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.567999    4882 generic.go:356] "Generic (PLEG): container finished" podID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerID="acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1" exitCode=0
Nov 01 04:22:29.568150 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.568045    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"
Nov 01 04:22:29.568150 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.568083    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" event={"ID":"2c4abcc8-b995-4851-9a00-5514e866ad89","Type":"ContainerDied","Data":"acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1"}
Nov 01 04:22:29.568281 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.568165    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt" event={"ID":"2c4abcc8-b995-4851-9a00-5514e866ad89","Type":"ContainerDied","Data":"6a5d15470ca7a1dc9e3b730c5be1344852c2d46c388e57d60209f892626afb61"}
Nov 01 04:22:29.568281 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.568202    4882 scope.go:122] "RemoveContainer" containerID="acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1"
Nov 01 04:22:29.580637 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.580619    4882 scope.go:122] "RemoveContainer" containerID="498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc"
Nov 01 04:22:29.590796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.590771    4882 scope.go:122] "RemoveContainer" containerID="acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1"
Nov 01 04:22:29.591132 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:22:29.591109    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1\": container with ID starting with acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1 not found: ID does not exist" containerID="acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1"
Nov 01 04:22:29.591211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.591140    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1"} err="failed to get container status \"acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1\": rpc error: code = NotFound desc = could not find container \"acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1\": container with ID starting with acefb03fbed17b927630a6ec74f7acb3537e590f21ac16cdffba94314f1b96e1 not found: ID does not exist"
Nov 01 04:22:29.591211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.591159    4882 scope.go:122] "RemoveContainer" containerID="498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc"
Nov 01 04:22:29.591440 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:22:29.591411    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc\": container with ID starting with 498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc not found: ID does not exist" containerID="498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc"
Nov 01 04:22:29.591508 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.591452    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc"} err="failed to get container status \"498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc\": rpc error: code = NotFound desc = could not find container \"498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc\": container with ID starting with 498d1593286670a583dab2f3ba7b385057d9f5802bdd5a30defca541226c2efc not found: ID does not exist"
Nov 01 04:22:29.602468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.602440    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9251b815-0f89-4f7c-bba6-046d34815b35" path="/var/lib/kubelet/pods/9251b815-0f89-4f7c-bba6-046d34815b35/volumes"
Nov 01 04:22:29.602979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.602958    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f526fe0f-78ff-45d9-9820-be2cbb1fe10e" path="/var/lib/kubelet/pods/f526fe0f-78ff-45d9-9820-be2cbb1fe10e/volumes"
Nov 01 04:22:29.793619 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.793476    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"]
Nov 01 04:22:29.806217 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:29.806179    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-131718-openshift-default-567c94bd9-f2gnt"]
Nov 01 04:22:31.601747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:31.601713    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" path="/var/lib/kubelet/pods/2c4abcc8-b995-4851-9a00-5514e866ad89/volumes"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.872830    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"]
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.873844    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.873865    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.873902    4882 cpu_manager.go:414] "containerMap: removing container" podUID="51b0ae48-6f64-4968-9931-e636eb4fb1d1" containerName="revoke-keys"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.873912    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b0ae48-6f64-4968-9931-e636eb4fb1d1" containerName="revoke-keys"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.873924    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.873932    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.874008    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9251b815-0f89-4f7c-bba6-046d34815b35" containerName="payload-processing"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.874016    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9251b815-0f89-4f7c-bba6-046d34815b35" containerName="payload-processing"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.874210    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.874227    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b0ae48-6f64-4968-9931-e636eb4fb1d1" containerName="revoke-keys"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.874240    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4abcc8-b995-4851-9a00-5514e866ad89" containerName="istio-proxy"
Nov 01 04:22:33.875461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.874269    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9251b815-0f89-4f7c-bba6-046d34815b35" containerName="payload-processing"
Nov 01 04:22:33.888534 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.887077    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:33.925906 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.925870    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:33.926209 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.926184    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gtcs\" (UniqueName: \"kubernetes.io/projected/134b18a1-1f73-4e97-b96a-aa0228239091-kube-api-access-8gtcs\") pod \"maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:33.926408 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.926393    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:33.929169 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:33.929143    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"]
Nov 01 04:22:34.002185 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:34.000100    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:22:34.002185 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:34.000263    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:22:34.028195 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:34.028157    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:34.028381 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:34.028258    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gtcs\" (UniqueName: \"kubernetes.io/projected/134b18a1-1f73-4e97-b96a-aa0228239091-kube-api-access-8gtcs\") pod \"maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:34.028381 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:34.028351    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:34.029492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:34.029463    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:34.031554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:34.031529    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:34.090286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:34.090198    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gtcs\" (UniqueName: \"kubernetes.io/projected/134b18a1-1f73-4e97-b96a-aa0228239091-kube-api-access-8gtcs\") pod \"maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:34.217278 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:34.217246    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:34.607850 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:34.607827    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"]
Nov 01 04:22:34.610379 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:22:34.610353    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134b18a1_1f73_4e97_b96a_aa0228239091.slice/crio-8960b3b018c28a556d196ec24e5740f738daec20c714176799afc1c1b8ee60ae WatchSource:0}: Error finding container 8960b3b018c28a556d196ec24e5740f738daec20c714176799afc1c1b8ee60ae: Status 404 returned error can't find the container with id 8960b3b018c28a556d196ec24e5740f738daec20c714176799afc1c1b8ee60ae
Nov 01 04:22:35.621146 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:35.621102    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs" event={"ID":"134b18a1-1f73-4e97-b96a-aa0228239091","Type":"ContainerStarted","Data":"daccfe94fac8699d0263f3494eeeb5bc726a6f8b5a5a2d54bec4b44321ff53ab"}
Nov 01 04:22:35.621556 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:35.621164    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs" event={"ID":"134b18a1-1f73-4e97-b96a-aa0228239091","Type":"ContainerStarted","Data":"8960b3b018c28a556d196ec24e5740f738daec20c714176799afc1c1b8ee60ae"}
Nov 01 04:22:35.741740 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:35.741685    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs" podStartSLOduration=2.741669634 podStartE2EDuration="2.741669634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:22:35.740505433 +0000 UTC m=+2424.302538452" watchObservedRunningTime="2026-11-01 04:22:35.741669634 +0000 UTC m=+2424.303702669"
Nov 01 04:22:36.594239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:36.594192    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"]
Nov 01 04:22:36.594498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:36.594407    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy" containerID="cri-o://96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26" gracePeriod=30
Nov 01 04:22:36.629468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:36.629411    4882 generic.go:356] "Generic (PLEG): container finished" podID="134b18a1-1f73-4e97-b96a-aa0228239091" containerID="daccfe94fac8699d0263f3494eeeb5bc726a6f8b5a5a2d54bec4b44321ff53ab" exitCode=0
Nov 01 04:22:36.629842 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:36.629503    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs" event={"ID":"134b18a1-1f73-4e97-b96a-aa0228239091","Type":"ContainerDied","Data":"daccfe94fac8699d0263f3494eeeb5bc726a6f8b5a5a2d54bec4b44321ff53ab"}
Nov 01 04:22:37.447570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.447536    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"]
Nov 01 04:22:37.451778 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.451757    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:37.505265 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.505232    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"]
Nov 01 04:22:37.586571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.586537    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp2ld\" (UniqueName: \"kubernetes.io/projected/af22a73b-7ab9-4f7e-aad9-777371ac110b-kube-api-access-hp2ld\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:37.586759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.586679    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:37.586841 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.586785    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:37.688699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.688662    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp2ld\" (UniqueName: \"kubernetes.io/projected/af22a73b-7ab9-4f7e-aad9-777371ac110b-kube-api-access-hp2ld\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:37.689173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.688778    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:37.689173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.688823    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:37.689572 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.689551    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:37.691651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.691624    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:37.746169 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.746140    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp2ld\" (UniqueName: \"kubernetes.io/projected/af22a73b-7ab9-4f7e-aad9-777371ac110b-kube-api-access-hp2ld\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:37.766522 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.766484    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:37.825606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.825338    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:37.890902 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.890868    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-openshift-ca\") pod \"134b18a1-1f73-4e97-b96a-aa0228239091\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") "
Nov 01 04:22:37.891056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.891010    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-xks-ca\") pod \"134b18a1-1f73-4e97-b96a-aa0228239091\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") "
Nov 01 04:22:37.891151 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.891136    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134b18a1-1f73-4e97-b96a-aa0228239091-kube-api-access-8gtcs\" (UniqueName: \"kubernetes.io/projected/134b18a1-1f73-4e97-b96a-aa0228239091-kube-api-access-8gtcs\") pod \"134b18a1-1f73-4e97-b96a-aa0228239091\" (UID: \"134b18a1-1f73-4e97-b96a-aa0228239091\") "
Nov 01 04:22:37.891423 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.891393    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-openshift-ca" pod "134b18a1-1f73-4e97-b96a-aa0228239091" (UID: "134b18a1-1f73-4e97-b96a-aa0228239091"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:22:37.893583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.893560    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134b18a1-1f73-4e97-b96a-aa0228239091-kube-api-access-8gtcs" pod "134b18a1-1f73-4e97-b96a-aa0228239091" (UID: "134b18a1-1f73-4e97-b96a-aa0228239091"). InnerVolumeSpecName "kube-api-access-8gtcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:22:37.899076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.899044    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-xks-ca" pod "134b18a1-1f73-4e97-b96a-aa0228239091" (UID: "134b18a1-1f73-4e97-b96a-aa0228239091"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:22:37.992493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.992400    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gtcs\" (UniqueName: \"kubernetes.io/projected/134b18a1-1f73-4e97-b96a-aa0228239091-kube-api-access-8gtcs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:37.992493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.992445    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:37.992493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:37.992460    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134b18a1-1f73-4e97-b96a-aa0228239091-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:38.150014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:38.149981    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"]
Nov 01 04:22:38.155286 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:22:38.155259    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf22a73b_7ab9_4f7e_aad9_777371ac110b.slice/crio-5a910d0d9aaded12080448b70ad244ccccdde7f3ea87265c8177dc6f4702d036 WatchSource:0}: Error finding container 5a910d0d9aaded12080448b70ad244ccccdde7f3ea87265c8177dc6f4702d036: Status 404 returned error can't find the container with id 5a910d0d9aaded12080448b70ad244ccccdde7f3ea87265c8177dc6f4702d036
Nov 01 04:22:38.643857 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:38.643767    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r" event={"ID":"af22a73b-7ab9-4f7e-aad9-777371ac110b","Type":"ContainerStarted","Data":"2437d5f4b39180ce672644007cc5b1f6d788c07d1284ffc1ddf564020157b5eb"}
Nov 01 04:22:38.643857 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:38.643832    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r" event={"ID":"af22a73b-7ab9-4f7e-aad9-777371ac110b","Type":"ContainerStarted","Data":"5a910d0d9aaded12080448b70ad244ccccdde7f3ea87265c8177dc6f4702d036"}
Nov 01 04:22:38.645528 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:38.645506    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"
Nov 01 04:22:38.645528 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:38.645512    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs" event={"ID":"134b18a1-1f73-4e97-b96a-aa0228239091","Type":"ContainerDied","Data":"8960b3b018c28a556d196ec24e5740f738daec20c714176799afc1c1b8ee60ae"}
Nov 01 04:22:38.645677 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:38.645555    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8960b3b018c28a556d196ec24e5740f738daec20c714176799afc1c1b8ee60ae"
Nov 01 04:22:38.751736 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:38.751626    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r" podStartSLOduration=1.751612453 podStartE2EDuration="1.751612453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:22:38.751176597 +0000 UTC m=+2427.313209628" watchObservedRunningTime="2026-11-01 04:22:38.751612453 +0000 UTC m=+2427.313645478"
Nov 01 04:22:39.653963 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:39.653926    4882 generic.go:356] "Generic (PLEG): container finished" podID="af22a73b-7ab9-4f7e-aad9-777371ac110b" containerID="2437d5f4b39180ce672644007cc5b1f6d788c07d1284ffc1ddf564020157b5eb" exitCode=0
Nov 01 04:22:39.654144 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:39.654010    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r" event={"ID":"af22a73b-7ab9-4f7e-aad9-777371ac110b","Type":"ContainerDied","Data":"2437d5f4b39180ce672644007cc5b1f6d788c07d1284ffc1ddf564020157b5eb"}
Nov 01 04:22:40.661613 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.661512    4882 generic.go:356] "Generic (PLEG): container finished" podID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerID="0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595" exitCode=0
Nov 01 04:22:40.661613 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.661585    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" event={"ID":"81ce7003-30a0-4d13-a9be-cf28b38a1a7d","Type":"ContainerDied","Data":"0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595"}
Nov 01 04:22:40.662168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.661961    4882 scope.go:122] "RemoveContainer" containerID="0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595"
Nov 01 04:22:40.663788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.663758    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:22:40.663919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.663843    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:22:40.663919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.663891    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:22:40.842772 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.842738    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:40.928766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.928686    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-xks-ca\") pod \"af22a73b-7ab9-4f7e-aad9-777371ac110b\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") "
Nov 01 04:22:40.928917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.928777    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af22a73b-7ab9-4f7e-aad9-777371ac110b-kube-api-access-hp2ld\" (UniqueName: \"kubernetes.io/projected/af22a73b-7ab9-4f7e-aad9-777371ac110b-kube-api-access-hp2ld\") pod \"af22a73b-7ab9-4f7e-aad9-777371ac110b\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") "
Nov 01 04:22:40.928917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.928888    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-openshift-ca\") pod \"af22a73b-7ab9-4f7e-aad9-777371ac110b\" (UID: \"af22a73b-7ab9-4f7e-aad9-777371ac110b\") "
Nov 01 04:22:40.929260 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.929232    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-openshift-ca" pod "af22a73b-7ab9-4f7e-aad9-777371ac110b" (UID: "af22a73b-7ab9-4f7e-aad9-777371ac110b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:22:40.930811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.930788    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af22a73b-7ab9-4f7e-aad9-777371ac110b-kube-api-access-hp2ld" pod "af22a73b-7ab9-4f7e-aad9-777371ac110b" (UID: "af22a73b-7ab9-4f7e-aad9-777371ac110b"). InnerVolumeSpecName "kube-api-access-hp2ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:22:40.930983 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:40.930968    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-xks-ca" pod "af22a73b-7ab9-4f7e-aad9-777371ac110b" (UID: "af22a73b-7ab9-4f7e-aad9-777371ac110b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:22:41.030936 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:41.030894    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:41.030936 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:41.030933    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp2ld\" (UniqueName: \"kubernetes.io/projected/af22a73b-7ab9-4f7e-aad9-777371ac110b-kube-api-access-hp2ld\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:41.031149 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:41.030951    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af22a73b-7ab9-4f7e-aad9-777371ac110b-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:41.693086 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:41.693046    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" event={"ID":"81ce7003-30a0-4d13-a9be-cf28b38a1a7d","Type":"ContainerStarted","Data":"3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827"}
Nov 01 04:22:41.695251 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:41.695204    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"
Nov 01 04:22:41.695251 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:41.695215    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r" event={"ID":"af22a73b-7ab9-4f7e-aad9-777371ac110b","Type":"ContainerDied","Data":"5a910d0d9aaded12080448b70ad244ccccdde7f3ea87265c8177dc6f4702d036"}
Nov 01 04:22:41.695421 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:41.695256    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a910d0d9aaded12080448b70ad244ccccdde7f3ea87265c8177dc6f4702d036"
Nov 01 04:22:41.904188 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:41.904162    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:22:42.047293 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.047251    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-credential-socket\") pod \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") "
Nov 01 04:22:42.047570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.047540    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-certs\") pod \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") "
Nov 01 04:22:42.047674 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.047599    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-kube-api-access-xshzs\" (UniqueName: \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-kube-api-access-xshzs\") pod \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") "
Nov 01 04:22:42.047738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.047672    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-token\" (UniqueName: \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-token\") pod \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") "
Nov 01 04:22:42.047738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.047732    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-socket\") pod \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") "
Nov 01 04:22:42.047857 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.047768    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-data\") pod \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") "
Nov 01 04:22:42.047857 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.047849    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-envoy\") pod \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") "
Nov 01 04:22:42.047970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.047894    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istiod-ca-cert\") pod \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") "
Nov 01 04:22:42.047970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.047929    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-podinfo\") pod \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\" (UID: \"769e10a8-d82e-4a7c-aec7-b79eb2cf1223\") "
Nov 01 04:22:42.051111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.051081    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-credential-socket" pod "769e10a8-d82e-4a7c-aec7-b79eb2cf1223" (UID: "769e10a8-d82e-4a7c-aec7-b79eb2cf1223"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:22:42.051297 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.051178    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-socket" pod "769e10a8-d82e-4a7c-aec7-b79eb2cf1223" (UID: "769e10a8-d82e-4a7c-aec7-b79eb2cf1223"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:22:42.051533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.051396    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-data" pod "769e10a8-d82e-4a7c-aec7-b79eb2cf1223" (UID: "769e10a8-d82e-4a7c-aec7-b79eb2cf1223"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:22:42.051533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.051394    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-certs" pod "769e10a8-d82e-4a7c-aec7-b79eb2cf1223" (UID: "769e10a8-d82e-4a7c-aec7-b79eb2cf1223"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:22:42.051895 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.051872    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istiod-ca-cert" pod "769e10a8-d82e-4a7c-aec7-b79eb2cf1223" (UID: "769e10a8-d82e-4a7c-aec7-b79eb2cf1223"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:22:42.054216 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.053766    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-podinfo" pod "769e10a8-d82e-4a7c-aec7-b79eb2cf1223" (UID: "769e10a8-d82e-4a7c-aec7-b79eb2cf1223"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:22:42.054216 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.054058    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-kube-api-access-xshzs" pod "769e10a8-d82e-4a7c-aec7-b79eb2cf1223" (UID: "769e10a8-d82e-4a7c-aec7-b79eb2cf1223"). InnerVolumeSpecName "kube-api-access-xshzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:22:42.054816 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.054792    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-envoy" pod "769e10a8-d82e-4a7c-aec7-b79eb2cf1223" (UID: "769e10a8-d82e-4a7c-aec7-b79eb2cf1223"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:22:42.057098 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.057059    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-token" pod "769e10a8-d82e-4a7c-aec7-b79eb2cf1223" (UID: "769e10a8-d82e-4a7c-aec7-b79eb2cf1223"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:22:42.150269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.150188    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:42.150269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.150220    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:42.150269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.150230    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:42.150269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.150238    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:42.150269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.150247    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:42.150269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.150256    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xshzs\" (UniqueName: \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-kube-api-access-xshzs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:42.150269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.150264    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:42.150689 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.150274    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:42.150689 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.150287    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e10a8-d82e-4a7c-aec7-b79eb2cf1223-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:42.509039 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.509004    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:22:42.510609 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.510578    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 01 04:22:42.510750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.510630    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 01 04:22:42.703265 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.703227    4882 generic.go:356] "Generic (PLEG): container finished" podID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerID="96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26" exitCode=0
Nov 01 04:22:42.703727 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.703269    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" event={"ID":"769e10a8-d82e-4a7c-aec7-b79eb2cf1223","Type":"ContainerDied","Data":"96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26"}
Nov 01 04:22:42.703727 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.703326    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl" event={"ID":"769e10a8-d82e-4a7c-aec7-b79eb2cf1223","Type":"ContainerDied","Data":"b724052bd4952b909e0e85d2da18f1b2fa4a2649945c2ce6acfdf3b804541a5b"}
Nov 01 04:22:42.703727 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.703349    4882 scope.go:122] "RemoveContainer" containerID="96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26"
Nov 01 04:22:42.703727 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.703277    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"
Nov 01 04:22:42.713900 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.713771    4882 scope.go:122] "RemoveContainer" containerID="03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827"
Nov 01 04:22:42.723014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.722994    4882 scope.go:122] "RemoveContainer" containerID="96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26"
Nov 01 04:22:42.723274 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:22:42.723257    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26\": container with ID starting with 96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26 not found: ID does not exist" containerID="96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26"
Nov 01 04:22:42.723324 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.723280    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26"} err="failed to get container status \"96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26\": rpc error: code = NotFound desc = could not find container \"96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26\": container with ID starting with 96bd084c58cae467c7eb4471da31702c94d6a8afd869e201c0335cb8b5fa7c26 not found: ID does not exist"
Nov 01 04:22:42.723324 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.723297    4882 scope.go:122] "RemoveContainer" containerID="03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827"
Nov 01 04:22:42.723554 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:22:42.723536    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827\": container with ID starting with 03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827 not found: ID does not exist" containerID="03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827"
Nov 01 04:22:42.723616 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.723566    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827"} err="failed to get container status \"03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827\": rpc error: code = NotFound desc = could not find container \"03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827\": container with ID starting with 03e8fb4467512dd7812d555398f099a35c2c0c84d48d3ed7b0c563f0d3562827 not found: ID does not exist"
Nov 01 04:22:42.857381 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.857297    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"]
Nov 01 04:22:42.905055 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:42.905020    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-601a4b-openshift-default-665b98cf95-frdcl"]
Nov 01 04:22:43.509066 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:43.509035    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 01 04:22:43.509233 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:43.509084    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 01 04:22:43.553475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:43.553443    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:22:43.609753 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:43.609718    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" path="/var/lib/kubelet/pods/769e10a8-d82e-4a7c-aec7-b79eb2cf1223/volumes"
Nov 01 04:22:44.509209 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:44.509177    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 01 04:22:44.509656 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:44.509230    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 01 04:22:45.509699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:45.509665    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 01 04:22:45.510082 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:45.509722    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 01 04:22:46.509734 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:46.509697    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 01 04:22:46.510174 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:46.509753    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 01 04:22:47.513473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:47.513442    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:22:47.514586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:47.514563    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:22:56.528422 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.528383    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"]
Nov 01 04:22:56.528834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.528595    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k" podUID="3c6b22ce-68d5-4f70-9378-62d19b20c73c" containerName="discovery" containerID="cri-o://8c3ff016ae3e6b7fe1f467a885925c0ea69d6c6df18f61ca6a6300d7f342a6e4" gracePeriod=30
Nov 01 04:22:56.807883 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.807854    4882 generic.go:356] "Generic (PLEG): container finished" podID="3c6b22ce-68d5-4f70-9378-62d19b20c73c" containerID="8c3ff016ae3e6b7fe1f467a885925c0ea69d6c6df18f61ca6a6300d7f342a6e4" exitCode=0
Nov 01 04:22:56.808019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.807928    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k" event={"ID":"3c6b22ce-68d5-4f70-9378-62d19b20c73c","Type":"ContainerDied","Data":"8c3ff016ae3e6b7fe1f467a885925c0ea69d6c6df18f61ca6a6300d7f342a6e4"}
Nov 01 04:22:56.808019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.807980    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k" event={"ID":"3c6b22ce-68d5-4f70-9378-62d19b20c73c","Type":"ContainerDied","Data":"74a1d8fe74a49e693f4bd76405f980f6ed071269282dd8ba743346bc606c2e59"}
Nov 01 04:22:56.808019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.807999    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74a1d8fe74a49e693f4bd76405f980f6ed071269282dd8ba743346bc606c2e59"
Nov 01 04:22:56.817523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.817503    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:22:56.843787 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.843759    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6b22ce-68d5-4f70-9378-62d19b20c73c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6b22ce-68d5-4f70-9378-62d19b20c73c-local-certs\") pod \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") "
Nov 01 04:22:56.843967 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.843886    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-token\" (UniqueName: \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-token\") pod \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") "
Nov 01 04:22:56.843967 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.843907    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-kubeconfig\") pod \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") "
Nov 01 04:22:56.843967 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.843926    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-dns-cert\") pod \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") "
Nov 01 04:22:56.844147 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.843991    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-cacerts\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-cacerts\") pod \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") "
Nov 01 04:22:56.844147 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.844051    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-kube-api-access-595bk\" (UniqueName: \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-kube-api-access-595bk\") pod \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") "
Nov 01 04:22:56.844147 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.844089    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-ca-configmap\") pod \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\" (UID: \"3c6b22ce-68d5-4f70-9378-62d19b20c73c\") "
Nov 01 04:22:56.844540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.844512    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-ca-configmap" pod "3c6b22ce-68d5-4f70-9378-62d19b20c73c" (UID: "3c6b22ce-68d5-4f70-9378-62d19b20c73c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:22:56.844975 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.844913    4882 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-ca-configmap\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:56.846386 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.846357    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-kubeconfig" pod "3c6b22ce-68d5-4f70-9378-62d19b20c73c" (UID: "3c6b22ce-68d5-4f70-9378-62d19b20c73c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:22:56.846749 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.846676    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6b22ce-68d5-4f70-9378-62d19b20c73c-local-certs" pod "3c6b22ce-68d5-4f70-9378-62d19b20c73c" (UID: "3c6b22ce-68d5-4f70-9378-62d19b20c73c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:22:56.846749 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.846695    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-token" pod "3c6b22ce-68d5-4f70-9378-62d19b20c73c" (UID: "3c6b22ce-68d5-4f70-9378-62d19b20c73c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:22:56.846913 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.846771    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-dns-cert" pod "3c6b22ce-68d5-4f70-9378-62d19b20c73c" (UID: "3c6b22ce-68d5-4f70-9378-62d19b20c73c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:22:56.846913 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.846881    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-kube-api-access-595bk" pod "3c6b22ce-68d5-4f70-9378-62d19b20c73c" (UID: "3c6b22ce-68d5-4f70-9378-62d19b20c73c"). InnerVolumeSpecName "kube-api-access-595bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:22:56.847043 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.847022    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-cacerts" pod "3c6b22ce-68d5-4f70-9378-62d19b20c73c" (UID: "3c6b22ce-68d5-4f70-9378-62d19b20c73c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:22:56.946694 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.946659    4882 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6b22ce-68d5-4f70-9378-62d19b20c73c-local-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:56.946694 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.946688    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:56.946694 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.946698    4882 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-kubeconfig\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:56.946958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.946708    4882 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-istio-csr-dns-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:56.946958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.946719    4882 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c6b22ce-68d5-4f70-9378-62d19b20c73c-cacerts\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:56.946958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:56.946728    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-595bk\" (UniqueName: \"kubernetes.io/projected/3c6b22ce-68d5-4f70-9378-62d19b20c73c-kube-api-access-595bk\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:22:57.814335 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:57.814299    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"
Nov 01 04:22:57.968532 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:57.968499    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"]
Nov 01 04:22:58.014444 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:58.014401    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k295k"]
Nov 01 04:22:58.722861 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:22:58.722810    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:22:59.600589 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:22:59.600554    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6b22ce-68d5-4f70-9378-62d19b20c73c" path="/var/lib/kubelet/pods/3c6b22ce-68d5-4f70-9378-62d19b20c73c/volumes"
Nov 01 04:23:05.413592 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.413560    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"]
Nov 01 04:23:05.414108 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414093    4882 cpu_manager.go:414] "containerMap: removing container" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy"
Nov 01 04:23:05.414154 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414109    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy"
Nov 01 04:23:05.414154 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414123    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3c6b22ce-68d5-4f70-9378-62d19b20c73c" containerName="discovery"
Nov 01 04:23:05.414154 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414128    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6b22ce-68d5-4f70-9378-62d19b20c73c" containerName="discovery"
Nov 01 04:23:05.414154 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414147    4882 cpu_manager.go:414] "containerMap: removing container" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy"
Nov 01 04:23:05.414154 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414152    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy"
Nov 01 04:23:05.414306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414175    4882 cpu_manager.go:414] "containerMap: removing container" podUID="134b18a1-1f73-4e97-b96a-aa0228239091" containerName="revoke-keys"
Nov 01 04:23:05.414306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414180    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134b18a1-1f73-4e97-b96a-aa0228239091" containerName="revoke-keys"
Nov 01 04:23:05.414306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414189    4882 cpu_manager.go:414] "containerMap: removing container" podUID="af22a73b-7ab9-4f7e-aad9-777371ac110b" containerName="revoke-keys"
Nov 01 04:23:05.414306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414193    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af22a73b-7ab9-4f7e-aad9-777371ac110b" containerName="revoke-keys"
Nov 01 04:23:05.414306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414269    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy"
Nov 01 04:23:05.414306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414282    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="134b18a1-1f73-4e97-b96a-aa0228239091" containerName="revoke-keys"
Nov 01 04:23:05.414306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414288    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="af22a73b-7ab9-4f7e-aad9-777371ac110b" containerName="revoke-keys"
Nov 01 04:23:05.414306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.414298    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6b22ce-68d5-4f70-9378-62d19b20c73c" containerName="discovery"
Nov 01 04:23:05.417848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.417824    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:05.514493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.514463    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:23:05.514493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.514463    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:23:05.519457 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.519420    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"]
Nov 01 04:23:05.544708 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.544686    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:05.544839 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.544819    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:05.544970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.544957    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmvgt\" (UniqueName: \"kubernetes.io/projected/cc2300af-4e61-4605-a5f2-70ab36ee23eb-kube-api-access-nmvgt\") pod \"maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:05.646397 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.646362    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:05.646606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.646511    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:05.646696 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.646662    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmvgt\" (UniqueName: \"kubernetes.io/projected/cc2300af-4e61-4605-a5f2-70ab36ee23eb-kube-api-access-nmvgt\") pod \"maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:05.647239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.647215    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:05.648742 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.648719    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:05.703884 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.703856    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmvgt\" (UniqueName: \"kubernetes.io/projected/cc2300af-4e61-4605-a5f2-70ab36ee23eb-kube-api-access-nmvgt\") pod \"maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:05.731114 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:05.731086    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:06.108202 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:06.108174    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"]
Nov 01 04:23:06.109386 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:23:06.109356    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2300af_4e61_4605_a5f2_70ab36ee23eb.slice/crio-e65b9003a8ba1e8d416656383bd2c7ca150a4600c30269306d883db544e64231 WatchSource:0}: Error finding container e65b9003a8ba1e8d416656383bd2c7ca150a4600c30269306d883db544e64231: Status 404 returned error can't find the container with id e65b9003a8ba1e8d416656383bd2c7ca150a4600c30269306d883db544e64231
Nov 01 04:23:06.876895 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:06.876862    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz" event={"ID":"cc2300af-4e61-4605-a5f2-70ab36ee23eb","Type":"ContainerStarted","Data":"10e8026e43ecf9afe8f01a1ea7d1daf0d5cc59913915870514edf03ddd57bab9"}
Nov 01 04:23:06.877314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:06.876915    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz" event={"ID":"cc2300af-4e61-4605-a5f2-70ab36ee23eb","Type":"ContainerStarted","Data":"e65b9003a8ba1e8d416656383bd2c7ca150a4600c30269306d883db544e64231"}
Nov 01 04:23:06.989150 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:06.989097    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz" podStartSLOduration=1.989082671 podStartE2EDuration="1.989082671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:23:06.987988387 +0000 UTC m=+2455.550021396" watchObservedRunningTime="2026-11-01 04:23:06.989082671 +0000 UTC m=+2455.551115687"
Nov 01 04:23:07.884890 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:07.884850    4882 generic.go:356] "Generic (PLEG): container finished" podID="cc2300af-4e61-4605-a5f2-70ab36ee23eb" containerID="10e8026e43ecf9afe8f01a1ea7d1daf0d5cc59913915870514edf03ddd57bab9" exitCode=0
Nov 01 04:23:07.885313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:07.884938    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz" event={"ID":"cc2300af-4e61-4605-a5f2-70ab36ee23eb","Type":"ContainerDied","Data":"10e8026e43ecf9afe8f01a1ea7d1daf0d5cc59913915870514edf03ddd57bab9"}
Nov 01 04:23:09.051942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.051915    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:09.192908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.192869    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-xks-ca\") pod \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") "
Nov 01 04:23:09.193110 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.192968    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2300af-4e61-4605-a5f2-70ab36ee23eb-kube-api-access-nmvgt\" (UniqueName: \"kubernetes.io/projected/cc2300af-4e61-4605-a5f2-70ab36ee23eb-kube-api-access-nmvgt\") pod \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") "
Nov 01 04:23:09.193110 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.193035    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-openshift-ca\") pod \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\" (UID: \"cc2300af-4e61-4605-a5f2-70ab36ee23eb\") "
Nov 01 04:23:09.193446 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.193399    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-openshift-ca" pod "cc2300af-4e61-4605-a5f2-70ab36ee23eb" (UID: "cc2300af-4e61-4605-a5f2-70ab36ee23eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:23:09.193631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.193612    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:09.195021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.194996    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-xks-ca" pod "cc2300af-4e61-4605-a5f2-70ab36ee23eb" (UID: "cc2300af-4e61-4605-a5f2-70ab36ee23eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:23:09.195548 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.195525    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2300af-4e61-4605-a5f2-70ab36ee23eb-kube-api-access-nmvgt" pod "cc2300af-4e61-4605-a5f2-70ab36ee23eb" (UID: "cc2300af-4e61-4605-a5f2-70ab36ee23eb"). InnerVolumeSpecName "kube-api-access-nmvgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:23:09.294546 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.294506    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2300af-4e61-4605-a5f2-70ab36ee23eb-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:09.294546 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.294542    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmvgt\" (UniqueName: \"kubernetes.io/projected/cc2300af-4e61-4605-a5f2-70ab36ee23eb-kube-api-access-nmvgt\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:09.901049 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.900959    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz" event={"ID":"cc2300af-4e61-4605-a5f2-70ab36ee23eb","Type":"ContainerDied","Data":"e65b9003a8ba1e8d416656383bd2c7ca150a4600c30269306d883db544e64231"}
Nov 01 04:23:09.901049 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.900989    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"
Nov 01 04:23:09.901049 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:09.901008    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e65b9003a8ba1e8d416656383bd2c7ca150a4600c30269306d883db544e64231"
Nov 01 04:23:13.436021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:13.435988    4882 scope.go:122] "RemoveContainer" containerID="a33071fec723ca0c7fe1b1ae6b9f77892281915bc8d19da81296f16100d3264e"
Nov 01 04:23:13.448132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:13.448112    4882 scope.go:122] "RemoveContainer" containerID="e379027c80a1f90e23008a89f71965a2b25c58376e74cd704fac3711039b477c"
Nov 01 04:23:13.457222 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:13.457201    4882 scope.go:122] "RemoveContainer" containerID="c6452093e1f44d7020502ae04d5b75722c12c77a2c98573a775c80e5c47cd80e"
Nov 01 04:23:13.466485 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:13.466466    4882 scope.go:122] "RemoveContainer" containerID="87ae0c7f5a00f62895b0c7a8c1fcf7c1bc4b760d27069c8199f04c8bcb846d31"
Nov 01 04:23:13.477078 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:13.477060    4882 scope.go:122] "RemoveContainer" containerID="8c3ff016ae3e6b7fe1f467a885925c0ea69d6c6df18f61ca6a6300d7f342a6e4"
Nov 01 04:23:28.723128 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:23:28.723036    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:23:29.725804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.725771    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"]
Nov 01 04:23:29.726321 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.726307    4882 cpu_manager.go:414] "containerMap: removing container" podUID="cc2300af-4e61-4605-a5f2-70ab36ee23eb" containerName="revoke-keys"
Nov 01 04:23:29.726373 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.726321    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2300af-4e61-4605-a5f2-70ab36ee23eb" containerName="revoke-keys"
Nov 01 04:23:29.726417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.726407    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2300af-4e61-4605-a5f2-70ab36ee23eb" containerName="revoke-keys"
Nov 01 04:23:29.726468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.726423    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="769e10a8-d82e-4a7c-aec7-b79eb2cf1223" containerName="istio-proxy"
Nov 01 04:23:29.731370 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.731350    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:29.780564 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.780534    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:23:29.784450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.784414    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"]
Nov 01 04:23:29.820365 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.820329    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwnqr\" (UniqueName: \"kubernetes.io/projected/772e9836-4f5f-4f97-b356-3084cdb1c359-kube-api-access-kwnqr\") pod \"maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:29.820548 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.820386    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:29.820627 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.820595    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:29.826606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.826585    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:23:29.921231 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.921192    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwnqr\" (UniqueName: \"kubernetes.io/projected/772e9836-4f5f-4f97-b356-3084cdb1c359-kube-api-access-kwnqr\") pod \"maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:29.921397 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.921249    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:29.921498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.921462    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:29.922044 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.922022    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:29.923604 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.923585    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:29.978078 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:29.978013    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwnqr\" (UniqueName: \"kubernetes.io/projected/772e9836-4f5f-4f97-b356-3084cdb1c359-kube-api-access-kwnqr\") pod \"maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:30.044240 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:30.044214    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:30.412116 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:30.412092    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"]
Nov 01 04:23:30.414093 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:23:30.414064    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772e9836_4f5f_4f97_b356_3084cdb1c359.slice/crio-d29460d6cbf2f9602b2ebf108a93134620d48861b3a4ffafb65d49b5f46d8750 WatchSource:0}: Error finding container d29460d6cbf2f9602b2ebf108a93134620d48861b3a4ffafb65d49b5f46d8750: Status 404 returned error can't find the container with id d29460d6cbf2f9602b2ebf108a93134620d48861b3a4ffafb65d49b5f46d8750
Nov 01 04:23:31.036285 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:31.036247    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x" event={"ID":"772e9836-4f5f-4f97-b356-3084cdb1c359","Type":"ContainerStarted","Data":"d58071b6faa933c059724ebf79be55b449e1d334154dfb83ac2b37e1a826b855"}
Nov 01 04:23:31.036685 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:31.036327    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x" event={"ID":"772e9836-4f5f-4f97-b356-3084cdb1c359","Type":"ContainerStarted","Data":"d29460d6cbf2f9602b2ebf108a93134620d48861b3a4ffafb65d49b5f46d8750"}
Nov 01 04:23:31.145017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:31.144962    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x" podStartSLOduration=2.144947748 podStartE2EDuration="2.144947748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:23:31.143459641 +0000 UTC m=+2479.705492658" watchObservedRunningTime="2026-11-01 04:23:31.144947748 +0000 UTC m=+2479.706980765"
Nov 01 04:23:32.044246 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:32.044211    4882 generic.go:356] "Generic (PLEG): container finished" podID="772e9836-4f5f-4f97-b356-3084cdb1c359" containerID="d58071b6faa933c059724ebf79be55b449e1d334154dfb83ac2b37e1a826b855" exitCode=0
Nov 01 04:23:32.044711 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:32.044296    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x" event={"ID":"772e9836-4f5f-4f97-b356-3084cdb1c359","Type":"ContainerDied","Data":"d58071b6faa933c059724ebf79be55b449e1d334154dfb83ac2b37e1a826b855"}
Nov 01 04:23:33.205769 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:33.205744    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:33.262728 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:33.262702    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772e9836-4f5f-4f97-b356-3084cdb1c359-kube-api-access-kwnqr\" (UniqueName: \"kubernetes.io/projected/772e9836-4f5f-4f97-b356-3084cdb1c359-kube-api-access-kwnqr\") pod \"772e9836-4f5f-4f97-b356-3084cdb1c359\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") "
Nov 01 04:23:33.262904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:33.262738    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-openshift-ca\") pod \"772e9836-4f5f-4f97-b356-3084cdb1c359\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") "
Nov 01 04:23:33.262904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:33.262854    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-xks-ca\") pod \"772e9836-4f5f-4f97-b356-3084cdb1c359\" (UID: \"772e9836-4f5f-4f97-b356-3084cdb1c359\") "
Nov 01 04:23:33.263112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:33.263088    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-openshift-ca" pod "772e9836-4f5f-4f97-b356-3084cdb1c359" (UID: "772e9836-4f5f-4f97-b356-3084cdb1c359"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:23:33.263376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:33.263358    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:33.264824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:33.264802    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-xks-ca" pod "772e9836-4f5f-4f97-b356-3084cdb1c359" (UID: "772e9836-4f5f-4f97-b356-3084cdb1c359"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:23:33.265010 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:33.264995    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772e9836-4f5f-4f97-b356-3084cdb1c359-kube-api-access-kwnqr" pod "772e9836-4f5f-4f97-b356-3084cdb1c359" (UID: "772e9836-4f5f-4f97-b356-3084cdb1c359"). InnerVolumeSpecName "kube-api-access-kwnqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:23:33.364172 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:33.364096    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwnqr\" (UniqueName: \"kubernetes.io/projected/772e9836-4f5f-4f97-b356-3084cdb1c359-kube-api-access-kwnqr\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:33.364172 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:33.364127    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772e9836-4f5f-4f97-b356-3084cdb1c359-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:34.058897 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:34.058866    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"
Nov 01 04:23:34.059082 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:34.058907    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x" event={"ID":"772e9836-4f5f-4f97-b356-3084cdb1c359","Type":"ContainerDied","Data":"d29460d6cbf2f9602b2ebf108a93134620d48861b3a4ffafb65d49b5f46d8750"}
Nov 01 04:23:34.059082 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:34.058963    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d29460d6cbf2f9602b2ebf108a93134620d48861b3a4ffafb65d49b5f46d8750"
Nov 01 04:23:41.172527 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:41.172483    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"]
Nov 01 04:23:41.218161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:41.218121    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3a348f-rb8rs"]
Nov 01 04:23:41.599638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:41.599597    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134b18a1-1f73-4e97-b96a-aa0228239091" path="/var/lib/kubelet/pods/134b18a1-1f73-4e97-b96a-aa0228239091/volumes"
Nov 01 04:23:48.136069 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:48.136035    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"]
Nov 01 04:23:48.136607 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:48.136259    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy" containerID="cri-o://f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393" gracePeriod=30
Nov 01 04:23:49.064120 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.064089    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"]
Nov 01 04:23:49.064687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.064671    4882 cpu_manager.go:414] "containerMap: removing container" podUID="772e9836-4f5f-4f97-b356-3084cdb1c359" containerName="revoke-keys"
Nov 01 04:23:49.064739 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.064688    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772e9836-4f5f-4f97-b356-3084cdb1c359" containerName="revoke-keys"
Nov 01 04:23:49.064801 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.064791    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="772e9836-4f5f-4f97-b356-3084cdb1c359" containerName="revoke-keys"
Nov 01 04:23:49.069266 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.069248    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:49.118392 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.118357    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:23:49.122236 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.122197    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"]
Nov 01 04:23:49.164127 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.164100    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:23:49.175281 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.175253    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:49.175398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.175371    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gk6b\" (UniqueName: \"kubernetes.io/projected/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-kube-api-access-9gk6b\") pod \"maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:49.175504 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.175476    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:49.276589 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.276558    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:49.276771 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.276615    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gk6b\" (UniqueName: \"kubernetes.io/projected/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-kube-api-access-9gk6b\") pod \"maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:49.276771 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.276650    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:49.277383 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.277358    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:49.278951 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.278931    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:49.333255 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.333186    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gk6b\" (UniqueName: \"kubernetes.io/projected/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-kube-api-access-9gk6b\") pod \"maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:49.382948 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.382896    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:49.764158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:49.764134    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"]
Nov 01 04:23:49.765924 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:23:49.765895    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab11570_7a8b_4d4d_b64b_0fbb9985c00a.slice/crio-504e2010c3b9028db835459dd2cea43eed40429df15033d4a87895f25f795794 WatchSource:0}: Error finding container 504e2010c3b9028db835459dd2cea43eed40429df15033d4a87895f25f795794: Status 404 returned error can't find the container with id 504e2010c3b9028db835459dd2cea43eed40429df15033d4a87895f25f795794
Nov 01 04:23:50.166325 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:50.166286    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f" event={"ID":"bab11570-7a8b-4d4d-b64b-0fbb9985c00a","Type":"ContainerStarted","Data":"e3a743bb9288e666586cf196a10aefbc1b380f86db8bf9fd003baf379d1304c1"}
Nov 01 04:23:50.166835 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:50.166351    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f" event={"ID":"bab11570-7a8b-4d4d-b64b-0fbb9985c00a","Type":"ContainerStarted","Data":"504e2010c3b9028db835459dd2cea43eed40429df15033d4a87895f25f795794"}
Nov 01 04:23:50.276952 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:50.276905    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f" podStartSLOduration=1.2768928050000001 podStartE2EDuration="1.276892805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:23:50.273776879 +0000 UTC m=+2498.835809895" watchObservedRunningTime="2026-11-01 04:23:50.276892805 +0000 UTC m=+2498.838925821"
Nov 01 04:23:51.173968 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:51.173935    4882 generic.go:356] "Generic (PLEG): container finished" podID="bab11570-7a8b-4d4d-b64b-0fbb9985c00a" containerID="e3a743bb9288e666586cf196a10aefbc1b380f86db8bf9fd003baf379d1304c1" exitCode=0
Nov 01 04:23:51.174366 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:51.174004    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f" event={"ID":"bab11570-7a8b-4d4d-b64b-0fbb9985c00a","Type":"ContainerDied","Data":"e3a743bb9288e666586cf196a10aefbc1b380f86db8bf9fd003baf379d1304c1"}
Nov 01 04:23:52.343711 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:52.343689    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:52.415684 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:52.415659    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-xks-ca\") pod \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") "
Nov 01 04:23:52.415846 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:52.415725    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-kube-api-access-9gk6b\" (UniqueName: \"kubernetes.io/projected/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-kube-api-access-9gk6b\") pod \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") "
Nov 01 04:23:52.415846 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:52.415754    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-openshift-ca\") pod \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\" (UID: \"bab11570-7a8b-4d4d-b64b-0fbb9985c00a\") "
Nov 01 04:23:52.416109 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:52.416078    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-openshift-ca" pod "bab11570-7a8b-4d4d-b64b-0fbb9985c00a" (UID: "bab11570-7a8b-4d4d-b64b-0fbb9985c00a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:23:52.416275 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:52.416258    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:52.417819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:52.417796    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-xks-ca" pod "bab11570-7a8b-4d4d-b64b-0fbb9985c00a" (UID: "bab11570-7a8b-4d4d-b64b-0fbb9985c00a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:23:52.417898 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:52.417866    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-kube-api-access-9gk6b" pod "bab11570-7a8b-4d4d-b64b-0fbb9985c00a" (UID: "bab11570-7a8b-4d4d-b64b-0fbb9985c00a"). InnerVolumeSpecName "kube-api-access-9gk6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:23:52.517986 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:52.517948    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gk6b\" (UniqueName: \"kubernetes.io/projected/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-kube-api-access-9gk6b\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:52.517986 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:52.517979    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab11570-7a8b-4d4d-b64b-0fbb9985c00a-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:53.188285 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.188255    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"
Nov 01 04:23:53.188444 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.188257    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f" event={"ID":"bab11570-7a8b-4d4d-b64b-0fbb9985c00a","Type":"ContainerDied","Data":"504e2010c3b9028db835459dd2cea43eed40429df15033d4a87895f25f795794"}
Nov 01 04:23:53.188444 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.188377    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="504e2010c3b9028db835459dd2cea43eed40429df15033d4a87895f25f795794"
Nov 01 04:23:53.408043 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.407986    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:23:53.530167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530139    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-token\" (UniqueName: \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-token\") pod \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") "
Nov 01 04:23:53.530345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530176    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-data\") pod \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") "
Nov 01 04:23:53.530345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530197    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-kube-api-access-m5qbf\" (UniqueName: \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-kube-api-access-m5qbf\") pod \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") "
Nov 01 04:23:53.530345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530243    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-credential-socket\") pod \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") "
Nov 01 04:23:53.530345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530297    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-envoy\") pod \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") "
Nov 01 04:23:53.530552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530364    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-certs\") pod \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") "
Nov 01 04:23:53.530552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530466    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istiod-ca-cert\") pod \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") "
Nov 01 04:23:53.530552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530502    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-data" pod "5804f1e8-7bd2-443d-89ef-8bb1389845d1" (UID: "5804f1e8-7bd2-443d-89ef-8bb1389845d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:23:53.530722 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530559    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-podinfo\") pod \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") "
Nov 01 04:23:53.530722 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530646    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-socket\") pod \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\" (UID: \"5804f1e8-7bd2-443d-89ef-8bb1389845d1\") "
Nov 01 04:23:53.530722 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530698    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-certs" pod "5804f1e8-7bd2-443d-89ef-8bb1389845d1" (UID: "5804f1e8-7bd2-443d-89ef-8bb1389845d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:23:53.530946 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.530911    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istiod-ca-cert" pod "5804f1e8-7bd2-443d-89ef-8bb1389845d1" (UID: "5804f1e8-7bd2-443d-89ef-8bb1389845d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:23:53.531037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.531016    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-credential-socket" pod "5804f1e8-7bd2-443d-89ef-8bb1389845d1" (UID: "5804f1e8-7bd2-443d-89ef-8bb1389845d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:23:53.531126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.531101    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-socket" pod "5804f1e8-7bd2-443d-89ef-8bb1389845d1" (UID: "5804f1e8-7bd2-443d-89ef-8bb1389845d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:23:53.531357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.531324    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:53.531497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.531364    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:53.531497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.531379    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:53.531497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.531394    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:53.531497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.531408    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:53.533922 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.533897    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-podinfo" pod "5804f1e8-7bd2-443d-89ef-8bb1389845d1" (UID: "5804f1e8-7bd2-443d-89ef-8bb1389845d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:23:53.534045 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.533938    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-kube-api-access-m5qbf" pod "5804f1e8-7bd2-443d-89ef-8bb1389845d1" (UID: "5804f1e8-7bd2-443d-89ef-8bb1389845d1"). InnerVolumeSpecName "kube-api-access-m5qbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:23:53.534243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.534224    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-envoy" pod "5804f1e8-7bd2-443d-89ef-8bb1389845d1" (UID: "5804f1e8-7bd2-443d-89ef-8bb1389845d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:23:53.535587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.535560    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-token" pod "5804f1e8-7bd2-443d-89ef-8bb1389845d1" (UID: "5804f1e8-7bd2-443d-89ef-8bb1389845d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:23:53.632889 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.632857    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:53.632889 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.632890    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:53.633063 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.632905    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qbf\" (UniqueName: \"kubernetes.io/projected/5804f1e8-7bd2-443d-89ef-8bb1389845d1-kube-api-access-m5qbf\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:53.633063 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:53.632922    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5804f1e8-7bd2-443d-89ef-8bb1389845d1-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:54.205358 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.205323    4882 generic.go:356] "Generic (PLEG): container finished" podID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerID="f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393" exitCode=0
Nov 01 04:23:54.205580 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.205380    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"
Nov 01 04:23:54.205580 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.205408    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" event={"ID":"5804f1e8-7bd2-443d-89ef-8bb1389845d1","Type":"ContainerDied","Data":"f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393"}
Nov 01 04:23:54.205580 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.205508    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq" event={"ID":"5804f1e8-7bd2-443d-89ef-8bb1389845d1","Type":"ContainerDied","Data":"5c8a49d6e174cd25b397fcd85c1e419dff3e6813b9408bf19a6d0174aea8a916"}
Nov 01 04:23:54.205580 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.205532    4882 scope.go:122] "RemoveContainer" containerID="f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393"
Nov 01 04:23:54.215483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.215455    4882 scope.go:122] "RemoveContainer" containerID="516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b"
Nov 01 04:23:54.224207 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.224192    4882 scope.go:122] "RemoveContainer" containerID="f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393"
Nov 01 04:23:54.224486 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:23:54.224465    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393\": container with ID starting with f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393 not found: ID does not exist" containerID="f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393"
Nov 01 04:23:54.224531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.224493    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393"} err="failed to get container status \"f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393\": rpc error: code = NotFound desc = could not find container \"f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393\": container with ID starting with f6bda7eae101ab7d77af6c01befbac09bc9b7670d16f408cb0e72bb2ff77c393 not found: ID does not exist"
Nov 01 04:23:54.224531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.224511    4882 scope.go:122] "RemoveContainer" containerID="516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b"
Nov 01 04:23:54.224755 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:23:54.224737    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b\": container with ID starting with 516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b not found: ID does not exist" containerID="516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b"
Nov 01 04:23:54.224797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.224762    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b"} err="failed to get container status \"516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b\": rpc error: code = NotFound desc = could not find container \"516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b\": container with ID starting with 516338e6411b1ab89f50a21ff30b108e337545ba6110cc218339016a5a43d65b not found: ID does not exist"
Nov 01 04:23:54.363148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.363117    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"]
Nov 01 04:23:54.409810 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:54.409779    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3a348f-openshift-default-7b46745bd5-r5knq"]
Nov 01 04:23:55.599791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:55.599756    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" path="/var/lib/kubelet/pods/5804f1e8-7bd2-443d-89ef-8bb1389845d1/volumes"
Nov 01 04:23:56.314915 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.314881    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"]
Nov 01 04:23:56.315116 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.315090    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6" podUID="7808dd74-e5e3-45c8-a8da-eb47fe119cfe" containerName="payload-processing" containerID="cri-o://ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049" gracePeriod=30
Nov 01 04:23:56.360651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.360623    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"]
Nov 01 04:23:56.360846 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.360825    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57" podUID="764e0ac7-3fa3-4fb7-a7c6-4592b8304930" containerName="payload-pre-processing" containerID="cri-o://ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa" gracePeriod=30
Nov 01 04:23:56.684190 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.684158    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:23:56.690398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.690377    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:23:56.772362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.772327    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-plugins-config-volume\") pod \"764e0ac7-3fa3-4fb7-a7c6-4592b8304930\" (UID: \"764e0ac7-3fa3-4fb7-a7c6-4592b8304930\") "
Nov 01 04:23:56.772571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.772380    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-kube-api-access-ncgnv\" (UniqueName: \"kubernetes.io/projected/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-kube-api-access-ncgnv\") pod \"7808dd74-e5e3-45c8-a8da-eb47fe119cfe\" (UID: \"7808dd74-e5e3-45c8-a8da-eb47fe119cfe\") "
Nov 01 04:23:56.772645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.772579    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-plugins-config-volume\") pod \"7808dd74-e5e3-45c8-a8da-eb47fe119cfe\" (UID: \"7808dd74-e5e3-45c8-a8da-eb47fe119cfe\") "
Nov 01 04:23:56.772645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.772640    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-kube-api-access-9b9dw\" (UniqueName: \"kubernetes.io/projected/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-kube-api-access-9b9dw\") pod \"764e0ac7-3fa3-4fb7-a7c6-4592b8304930\" (UID: \"764e0ac7-3fa3-4fb7-a7c6-4592b8304930\") "
Nov 01 04:23:56.772909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.772785    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-plugins-config-volume" pod "764e0ac7-3fa3-4fb7-a7c6-4592b8304930" (UID: "764e0ac7-3fa3-4fb7-a7c6-4592b8304930"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:23:56.773003 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.772983    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-plugins-config-volume" pod "7808dd74-e5e3-45c8-a8da-eb47fe119cfe" (UID: "7808dd74-e5e3-45c8-a8da-eb47fe119cfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:23:56.773166 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.773147    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:56.773225 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.773171    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:56.774583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.774565    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-kube-api-access-ncgnv" pod "7808dd74-e5e3-45c8-a8da-eb47fe119cfe" (UID: "7808dd74-e5e3-45c8-a8da-eb47fe119cfe"). InnerVolumeSpecName "kube-api-access-ncgnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:23:56.774777 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.774756    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-kube-api-access-9b9dw" pod "764e0ac7-3fa3-4fb7-a7c6-4592b8304930" (UID: "764e0ac7-3fa3-4fb7-a7c6-4592b8304930"). InnerVolumeSpecName "kube-api-access-9b9dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:23:56.898510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.894402    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b9dw\" (UniqueName: \"kubernetes.io/projected/764e0ac7-3fa3-4fb7-a7c6-4592b8304930-kube-api-access-9b9dw\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:56.898510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:56.894483    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncgnv\" (UniqueName: \"kubernetes.io/projected/7808dd74-e5e3-45c8-a8da-eb47fe119cfe-kube-api-access-ncgnv\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:23:57.227168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.227128    4882 generic.go:356] "Generic (PLEG): container finished" podID="764e0ac7-3fa3-4fb7-a7c6-4592b8304930" containerID="ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa" exitCode=0
Nov 01 04:23:57.227168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.227169    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"
Nov 01 04:23:57.227413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.227208    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57" event={"ID":"764e0ac7-3fa3-4fb7-a7c6-4592b8304930","Type":"ContainerDied","Data":"ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa"}
Nov 01 04:23:57.227413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.227258    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57" event={"ID":"764e0ac7-3fa3-4fb7-a7c6-4592b8304930","Type":"ContainerDied","Data":"4baab1503f05e34d5b39b7f277f1f8b27ebb0d64a679db069e5775e4846767e1"}
Nov 01 04:23:57.227413 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.227281    4882 scope.go:122] "RemoveContainer" containerID="ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa"
Nov 01 04:23:57.228751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.228729    4882 generic.go:356] "Generic (PLEG): container finished" podID="7808dd74-e5e3-45c8-a8da-eb47fe119cfe" containerID="ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049" exitCode=0
Nov 01 04:23:57.228868 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.228769    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"
Nov 01 04:23:57.228868 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.228818    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6" event={"ID":"7808dd74-e5e3-45c8-a8da-eb47fe119cfe","Type":"ContainerDied","Data":"ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049"}
Nov 01 04:23:57.228952 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.228880    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6" event={"ID":"7808dd74-e5e3-45c8-a8da-eb47fe119cfe","Type":"ContainerDied","Data":"aa34a11d185d8349f6dcaa757c6697d2a1ad3f283f7f0eef8c6f96f7c1a8660f"}
Nov 01 04:23:57.239070 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.239055    4882 scope.go:122] "RemoveContainer" containerID="ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa"
Nov 01 04:23:57.239529 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:23:57.239500    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa\": container with ID starting with ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa not found: ID does not exist" containerID="ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa"
Nov 01 04:23:57.239626 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.239537    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa"} err="failed to get container status \"ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa\": rpc error: code = NotFound desc = could not find container \"ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa\": container with ID starting with ebffffbb4ad3c28f4cf3dcee01862f4617737f87bc4a3cb8b75d9d13e0dc08aa not found: ID does not exist"
Nov 01 04:23:57.239626 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.239557    4882 scope.go:122] "RemoveContainer" containerID="ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049"
Nov 01 04:23:57.248684 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.248666    4882 scope.go:122] "RemoveContainer" containerID="ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049"
Nov 01 04:23:57.248921 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:23:57.248904    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049\": container with ID starting with ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049 not found: ID does not exist" containerID="ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049"
Nov 01 04:23:57.248968 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.248929    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049"} err="failed to get container status \"ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049\": rpc error: code = NotFound desc = could not find container \"ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049\": container with ID starting with ebfe67e514c1d78f78fd6e9e4e887a63a1663db8eb52251e13186a6ba7080049 not found: ID does not exist"
Nov 01 04:23:57.380389 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.380348    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"]
Nov 01 04:23:57.386607 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.386577    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-555d48bbc9-bdf57"]
Nov 01 04:23:57.487844 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.487743    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"]
Nov 01 04:23:57.492075 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.492049    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-75cf8487d-g24x6"]
Nov 01 04:23:57.601257 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.601225    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764e0ac7-3fa3-4fb7-a7c6-4592b8304930" path="/var/lib/kubelet/pods/764e0ac7-3fa3-4fb7-a7c6-4592b8304930/volumes"
Nov 01 04:23:57.601704 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:57.601689    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7808dd74-e5e3-45c8-a8da-eb47fe119cfe" path="/var/lib/kubelet/pods/7808dd74-e5e3-45c8-a8da-eb47fe119cfe/volumes"
Nov 01 04:23:58.197637 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:58.197607    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"]
Nov 01 04:23:58.203554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:58.203522    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a7b5f0-ztzg8"]
Nov 01 04:23:58.723388 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:23:58.723348    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:23:59.600421 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:23:59.600391    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b0ae48-6f64-4968-9931-e636eb4fb1d1" path="/var/lib/kubelet/pods/51b0ae48-6f64-4968-9931-e636eb4fb1d1/volumes"
Nov 01 04:24:04.870678 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:04.870638    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"]
Nov 01 04:24:04.918670 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:04.918631    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee0f8-vrw4r"]
Nov 01 04:24:05.588416 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:05.588380    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"]
Nov 01 04:24:05.588675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:05.588652    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy" containerID="cri-o://5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d" gracePeriod=30
Nov 01 04:24:05.600379 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:05.600349    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af22a73b-7ab9-4f7e-aad9-777371ac110b" path="/var/lib/kubelet/pods/af22a73b-7ab9-4f7e-aad9-777371ac110b/volumes"
Nov 01 04:24:06.193587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:06.193545    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"]
Nov 01 04:24:06.197113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:06.197084    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4c2a66-6mrpz"]
Nov 01 04:24:07.599981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.599938    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2300af-4e61-4605-a5f2-70ab36ee23eb" path="/var/lib/kubelet/pods/cc2300af-4e61-4605-a5f2-70ab36ee23eb/volumes"
Nov 01 04:24:07.765797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.765763    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"]
Nov 01 04:24:07.766292 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766278    4882 cpu_manager.go:414] "containerMap: removing container" podUID="764e0ac7-3fa3-4fb7-a7c6-4592b8304930" containerName="payload-pre-processing"
Nov 01 04:24:07.766357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766294    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764e0ac7-3fa3-4fb7-a7c6-4592b8304930" containerName="payload-pre-processing"
Nov 01 04:24:07.766357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766314    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy"
Nov 01 04:24:07.766357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766320    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy"
Nov 01 04:24:07.766357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766330    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7808dd74-e5e3-45c8-a8da-eb47fe119cfe" containerName="payload-processing"
Nov 01 04:24:07.766357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766335    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7808dd74-e5e3-45c8-a8da-eb47fe119cfe" containerName="payload-processing"
Nov 01 04:24:07.766357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766349    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bab11570-7a8b-4d4d-b64b-0fbb9985c00a" containerName="revoke-keys"
Nov 01 04:24:07.766357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766354    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab11570-7a8b-4d4d-b64b-0fbb9985c00a" containerName="revoke-keys"
Nov 01 04:24:07.766613 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766449    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab11570-7a8b-4d4d-b64b-0fbb9985c00a" containerName="revoke-keys"
Nov 01 04:24:07.766613 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766463    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy"
Nov 01 04:24:07.766613 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766473    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="764e0ac7-3fa3-4fb7-a7c6-4592b8304930" containerName="payload-pre-processing"
Nov 01 04:24:07.766613 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.766482    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7808dd74-e5e3-45c8-a8da-eb47fe119cfe" containerName="payload-processing"
Nov 01 04:24:07.770103 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.770081    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.818845 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.818815    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-550b8a-openshift-default-dockercfg-tlcvw\""
Nov 01 04:24:07.868997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.868911    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"]
Nov 01 04:24:07.882511 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.882476    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-socket\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.882705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.882540    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-data\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.882705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.882609    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b18693-924b-492c-9db8-29a52ef1bb12-istiod-ca-cert\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.882840 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.882703    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-istio-token\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.882840 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.882789    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtktw\" (UniqueName: \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-kube-api-access-rtktw\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.882953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.882916    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-envoy\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.883010 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.882978    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-certs\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.883111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.883089    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b18693-924b-492c-9db8-29a52ef1bb12-istio-podinfo\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.883178 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.883126    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-credential-socket\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.984298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.984262    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-data\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.984298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.984303    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b18693-924b-492c-9db8-29a52ef1bb12-istiod-ca-cert\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.984566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.984335    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-istio-token\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.984566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.984385    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtktw\" (UniqueName: \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-kube-api-access-rtktw\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.984566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.984496    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-envoy\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.984566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.984559    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-certs\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.984721 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.984623    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b18693-924b-492c-9db8-29a52ef1bb12-istio-podinfo\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.984721 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.984663    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-credential-socket\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.984798 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.984729    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-socket\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.984798 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.984746    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-data\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.985272 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.985246    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-socket\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.985833 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.985791    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-certs\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.986224 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.986198    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b18693-924b-492c-9db8-29a52ef1bb12-istiod-ca-cert\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.986500 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.986475    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-credential-socket\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.988258 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.988233    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-envoy\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:07.988536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:07.988515    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b18693-924b-492c-9db8-29a52ef1bb12-istio-podinfo\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:08.042559 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:08.042526    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-istio-token\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:08.042740 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:08.042572    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtktw\" (UniqueName: \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-kube-api-access-rtktw\") pod \"e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") " pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:08.084942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:08.084901    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:08.507568 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:08.507538    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"]
Nov 01 04:24:08.507717 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:24:08.507687    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b18693_924b_492c_9db8_29a52ef1bb12.slice/crio-622a4e1f62c0308f7410526ad4c7935079d21304120ce6a1c6e372243b36fbe4 WatchSource:0}: Error finding container 622a4e1f62c0308f7410526ad4c7935079d21304120ce6a1c6e372243b36fbe4: Status 404 returned error can't find the container with id 622a4e1f62c0308f7410526ad4c7935079d21304120ce6a1c6e372243b36fbe4
Nov 01 04:24:08.510331 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:08.510299    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:24:08.510456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:08.510371    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:24:08.510456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:08.510406    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:24:09.317590 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:09.317554    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" event={"ID":"71b18693-924b-492c-9db8-29a52ef1bb12","Type":"ContainerStarted","Data":"721cb0d78fb595f35fdcbda10a830219c40705e52ec9d66ab3afb43afe565a73"}
Nov 01 04:24:09.317984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:09.317617    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" event={"ID":"71b18693-924b-492c-9db8-29a52ef1bb12","Type":"ContainerStarted","Data":"622a4e1f62c0308f7410526ad4c7935079d21304120ce6a1c6e372243b36fbe4"}
Nov 01 04:24:09.525311 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:09.525260    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" podStartSLOduration=2.525245437 podStartE2EDuration="2.525245437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:24:09.475771796 +0000 UTC m=+2518.037804803" watchObservedRunningTime="2026-11-01 04:24:09.525245437 +0000 UTC m=+2518.087278453"
Nov 01 04:24:10.086023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:10.085985    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:10.091355 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:10.091327    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:10.324251 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:10.324212    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:10.325169 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:10.325149    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:24:10.870234 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:10.870173    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:24:11.019705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.019667    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-envoy\") pod \"7865f262-e4a2-4315-89c3-8e5655f1e47f\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") "
Nov 01 04:24:11.019890 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.019813    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-certs\") pod \"7865f262-e4a2-4315-89c3-8e5655f1e47f\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") "
Nov 01 04:24:11.019890 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.019869    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-token\" (UniqueName: \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-token\") pod \"7865f262-e4a2-4315-89c3-8e5655f1e47f\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") "
Nov 01 04:24:11.020026 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.019930    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-data\") pod \"7865f262-e4a2-4315-89c3-8e5655f1e47f\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") "
Nov 01 04:24:11.020090 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.020025    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-credential-socket\") pod \"7865f262-e4a2-4315-89c3-8e5655f1e47f\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") "
Nov 01 04:24:11.020090 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.020056    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-podinfo\") pod \"7865f262-e4a2-4315-89c3-8e5655f1e47f\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") "
Nov 01 04:24:11.020205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.020106    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-certs" pod "7865f262-e4a2-4315-89c3-8e5655f1e47f" (UID: "7865f262-e4a2-4315-89c3-8e5655f1e47f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:11.020205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.020118    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7865f262-e4a2-4315-89c3-8e5655f1e47f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7865f262-e4a2-4315-89c3-8e5655f1e47f-istiod-ca-cert\") pod \"7865f262-e4a2-4315-89c3-8e5655f1e47f\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") "
Nov 01 04:24:11.020205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.020179    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-kube-api-access-6cmv6\" (UniqueName: \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-kube-api-access-6cmv6\") pod \"7865f262-e4a2-4315-89c3-8e5655f1e47f\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") "
Nov 01 04:24:11.020495 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.020470    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7865f262-e4a2-4315-89c3-8e5655f1e47f-istiod-ca-cert" pod "7865f262-e4a2-4315-89c3-8e5655f1e47f" (UID: "7865f262-e4a2-4315-89c3-8e5655f1e47f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:24:11.020773 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.020644    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-socket" pod "7865f262-e4a2-4315-89c3-8e5655f1e47f" (UID: "7865f262-e4a2-4315-89c3-8e5655f1e47f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:11.021072 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.020613    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-socket\") pod \"7865f262-e4a2-4315-89c3-8e5655f1e47f\" (UID: \"7865f262-e4a2-4315-89c3-8e5655f1e47f\") "
Nov 01 04:24:11.021072 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.020894    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-credential-socket" pod "7865f262-e4a2-4315-89c3-8e5655f1e47f" (UID: "7865f262-e4a2-4315-89c3-8e5655f1e47f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:11.021072 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.020873    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-data" pod "7865f262-e4a2-4315-89c3-8e5655f1e47f" (UID: "7865f262-e4a2-4315-89c3-8e5655f1e47f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:11.021726 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.021703    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:11.022754 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.022729    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:11.022905 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.022756    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:11.022905 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.022771    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7865f262-e4a2-4315-89c3-8e5655f1e47f-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:11.022905 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.022290    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-envoy" pod "7865f262-e4a2-4315-89c3-8e5655f1e47f" (UID: "7865f262-e4a2-4315-89c3-8e5655f1e47f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:11.022905 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.022798    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:11.023244 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.023219    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-kube-api-access-6cmv6" pod "7865f262-e4a2-4315-89c3-8e5655f1e47f" (UID: "7865f262-e4a2-4315-89c3-8e5655f1e47f"). InnerVolumeSpecName "kube-api-access-6cmv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:24:11.023366 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.023347    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-token" pod "7865f262-e4a2-4315-89c3-8e5655f1e47f" (UID: "7865f262-e4a2-4315-89c3-8e5655f1e47f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:24:11.023405 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.023370    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-podinfo" pod "7865f262-e4a2-4315-89c3-8e5655f1e47f" (UID: "7865f262-e4a2-4315-89c3-8e5655f1e47f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:24:11.124211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.124142    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cmv6\" (UniqueName: \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-kube-api-access-6cmv6\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:11.124211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.124170    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:11.124211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.124179    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:11.124211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.124188    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7865f262-e4a2-4315-89c3-8e5655f1e47f-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:11.331480 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.331418    4882 generic.go:356] "Generic (PLEG): container finished" podID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerID="5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d" exitCode=0
Nov 01 04:24:11.331480 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.331457    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" event={"ID":"7865f262-e4a2-4315-89c3-8e5655f1e47f","Type":"ContainerDied","Data":"5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d"}
Nov 01 04:24:11.331986 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.331489    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"
Nov 01 04:24:11.331986 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.331517    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq" event={"ID":"7865f262-e4a2-4315-89c3-8e5655f1e47f","Type":"ContainerDied","Data":"95dc890b1d8643e23f08fbbf60fbcef7d05448afff42357c64052bb68b415eb7"}
Nov 01 04:24:11.331986 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.331540    4882 scope.go:122] "RemoveContainer" containerID="5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d"
Nov 01 04:24:11.342796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.342779    4882 scope.go:122] "RemoveContainer" containerID="30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e"
Nov 01 04:24:11.351931 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.351914    4882 scope.go:122] "RemoveContainer" containerID="5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d"
Nov 01 04:24:11.352178 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:24:11.352158    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d\": container with ID starting with 5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d not found: ID does not exist" containerID="5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d"
Nov 01 04:24:11.352226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.352186    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d"} err="failed to get container status \"5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d\": rpc error: code = NotFound desc = could not find container \"5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d\": container with ID starting with 5993e6adb69865039cf4d0477a5ddb58acee575a7478ab2d899362ea4856b93d not found: ID does not exist"
Nov 01 04:24:11.352226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.352205    4882 scope.go:122] "RemoveContainer" containerID="30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e"
Nov 01 04:24:11.352413 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:24:11.352399    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e\": container with ID starting with 30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e not found: ID does not exist" containerID="30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e"
Nov 01 04:24:11.352571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.352417    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e"} err="failed to get container status \"30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e\": rpc error: code = NotFound desc = could not find container \"30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e\": container with ID starting with 30598004e5420e397f464fe8d7b5476ee062d8474a991ee1c4ccf27f0294059e not found: ID does not exist"
Nov 01 04:24:11.487293 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.487258    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"]
Nov 01 04:24:11.494359 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.494332    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a7b5f0-openshift-default-9994757f5-z76cq"]
Nov 01 04:24:11.600183 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.600152    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" path="/var/lib/kubelet/pods/7865f262-e4a2-4315-89c3-8e5655f1e47f/volumes"
Nov 01 04:24:11.780335 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.780248    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"]
Nov 01 04:24:11.780597 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:11.780524    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy" containerID="cri-o://248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1" gracePeriod=30
Nov 01 04:24:13.712595 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:13.710318    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"]
Nov 01 04:24:13.712595 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:13.710689    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy" containerID="cri-o://ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7" gracePeriod=30
Nov 01 04:24:14.083003 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.082971    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"]
Nov 01 04:24:14.083485 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.083472    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy"
Nov 01 04:24:14.083526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.083487    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy"
Nov 01 04:24:14.083526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.083497    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy"
Nov 01 04:24:14.083526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.083503    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy"
Nov 01 04:24:14.083647 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.083633    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy"
Nov 01 04:24:14.083647 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.083645    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5804f1e8-7bd2-443d-89ef-8bb1389845d1" containerName="istio-proxy"
Nov 01 04:24:14.088556 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.088533    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.137540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.137507    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ad9e22-openshift-default-dockercfg-s6bsk\""
Nov 01 04:24:14.146393 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.146360    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"]
Nov 01 04:24:14.158227 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.158197    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-token\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.158378 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.158241    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-data\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.158378 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.158338    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znb8x\" (UniqueName: \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-kube-api-access-znb8x\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.158528 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.158418    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ca8add-0434-46a9-a055-0306697ac2d7-istiod-ca-cert\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.158528 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.158462    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-socket\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.158528 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.158497    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-envoy\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.158694 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.158591    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-credential-socket\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.158694 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.158624    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-certs\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.158802 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.158698    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-podinfo\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.270853 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.270816    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-credential-socket\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.271076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.270879    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-certs\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.271076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.270952    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-podinfo\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.271076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.271048    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-token\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.271275 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.271091    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-data\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.271275 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.271167    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znb8x\" (UniqueName: \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-kube-api-access-znb8x\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.271275 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.271226    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ca8add-0434-46a9-a055-0306697ac2d7-istiod-ca-cert\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.271275 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.271271    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-socket\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.271641 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.271309    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-envoy\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.272134 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.272085    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-certs\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.272717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.272667    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-credential-socket\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.273138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.273114    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-data\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.273652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.273627    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-socket\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.273808 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.273787    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ca8add-0434-46a9-a055-0306697ac2d7-istiod-ca-cert\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.275828 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.275803    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-envoy\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.282232 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.282209    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-podinfo\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.329137 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.329104    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-token\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.329296 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.329144    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znb8x\" (UniqueName: \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-kube-api-access-znb8x\") pod \"e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") " pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.401406 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.401318    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:14.878464 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.878415    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"]
Nov 01 04:24:14.880037 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:24:14.880009    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ca8add_0434_46a9_a055_0306697ac2d7.slice/crio-80361abc6f4da6a4a10e2271d267c46621cac99d38315016c15ef5338bb7368a WatchSource:0}: Error finding container 80361abc6f4da6a4a10e2271d267c46621cac99d38315016c15ef5338bb7368a: Status 404 returned error can't find the container with id 80361abc6f4da6a4a10e2271d267c46621cac99d38315016c15ef5338bb7368a
Nov 01 04:24:14.882597 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.882566    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:24:14.882671 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.882628    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:24:14.882671 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:14.882660    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:24:15.361572 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:15.361534    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" event={"ID":"d8ca8add-0434-46a9-a055-0306697ac2d7","Type":"ContainerStarted","Data":"a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714"}
Nov 01 04:24:15.361738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:15.361609    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" event={"ID":"d8ca8add-0434-46a9-a055-0306697ac2d7","Type":"ContainerStarted","Data":"80361abc6f4da6a4a10e2271d267c46621cac99d38315016c15ef5338bb7368a"}
Nov 01 04:24:15.402120 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:15.402088    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:15.408086 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:15.408057    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:15.477343 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:15.477288    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" podStartSLOduration=1.47727351 podStartE2EDuration="1.47727351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:24:15.474137722 +0000 UTC m=+2524.036170739" watchObservedRunningTime="2026-11-01 04:24:15.47727351 +0000 UTC m=+2524.039306526"
Nov 01 04:24:16.036038 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.036004    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"]
Nov 01 04:24:16.036555 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.036541    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy"
Nov 01 04:24:16.036555 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.036555    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy"
Nov 01 04:24:16.036683 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.036645    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7865f262-e4a2-4315-89c3-8e5655f1e47f" containerName="istio-proxy"
Nov 01 04:24:16.040194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.040173    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:16.088297 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.088263    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:24:16.088495 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.088411    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:24:16.146367 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.146329    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"]
Nov 01 04:24:16.201111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.201077    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:16.201294 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.201216    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h99jn\" (UniqueName: \"kubernetes.io/projected/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-kube-api-access-h99jn\") pod \"maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:16.201294 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.201282    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:16.302944 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.302855    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:16.303126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.302966    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h99jn\" (UniqueName: \"kubernetes.io/projected/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-kube-api-access-h99jn\") pod \"maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:16.303126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.303030    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:16.303665 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.303639    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:16.305247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.305226    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:16.360180 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.360148    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h99jn\" (UniqueName: \"kubernetes.io/projected/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-kube-api-access-h99jn\") pod \"maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:16.372014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.371985    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:16.373040 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.373023    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:24:16.654742 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:16.654658    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:17.030722 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.030698    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:24:17.032027 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.032005    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"]
Nov 01 04:24:17.033358 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:24:17.033329    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2d747f_37e6_4d64_ba8e_6b5bc33ac9ef.slice/crio-81425bc4c2c6865eda85d70e3159248820f043ef64b2ff0f5874a0351e222814 WatchSource:0}: Error finding container 81425bc4c2c6865eda85d70e3159248820f043ef64b2ff0f5874a0351e222814: Status 404 returned error can't find the container with id 81425bc4c2c6865eda85d70e3159248820f043ef64b2ff0f5874a0351e222814
Nov 01 04:24:17.113647 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.113614    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-kube-api-access-nb7nb\" (UniqueName: \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-kube-api-access-nb7nb\") pod \"c3109f72-8937-4f69-b7f0-25b2e4870336\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") "
Nov 01 04:24:17.114081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.113681    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-token\" (UniqueName: \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-token\") pod \"c3109f72-8937-4f69-b7f0-25b2e4870336\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") "
Nov 01 04:24:17.114081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.113713    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-podinfo\") pod \"c3109f72-8937-4f69-b7f0-25b2e4870336\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") "
Nov 01 04:24:17.114081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.113754    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-credential-socket\") pod \"c3109f72-8937-4f69-b7f0-25b2e4870336\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") "
Nov 01 04:24:17.114081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.113882    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-data\") pod \"c3109f72-8937-4f69-b7f0-25b2e4870336\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") "
Nov 01 04:24:17.114081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.113946    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-envoy\") pod \"c3109f72-8937-4f69-b7f0-25b2e4870336\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") "
Nov 01 04:24:17.114081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.113984    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3109f72-8937-4f69-b7f0-25b2e4870336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3109f72-8937-4f69-b7f0-25b2e4870336-istiod-ca-cert\") pod \"c3109f72-8937-4f69-b7f0-25b2e4870336\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") "
Nov 01 04:24:17.114472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114108    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-socket\") pod \"c3109f72-8937-4f69-b7f0-25b2e4870336\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") "
Nov 01 04:24:17.114472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114175    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-certs\") pod \"c3109f72-8937-4f69-b7f0-25b2e4870336\" (UID: \"c3109f72-8937-4f69-b7f0-25b2e4870336\") "
Nov 01 04:24:17.114472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114108    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-credential-socket" pod "c3109f72-8937-4f69-b7f0-25b2e4870336" (UID: "c3109f72-8937-4f69-b7f0-25b2e4870336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:17.114472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114143    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-data" pod "c3109f72-8937-4f69-b7f0-25b2e4870336" (UID: "c3109f72-8937-4f69-b7f0-25b2e4870336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:17.114472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114362    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-socket" pod "c3109f72-8937-4f69-b7f0-25b2e4870336" (UID: "c3109f72-8937-4f69-b7f0-25b2e4870336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:17.114753 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114469    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3109f72-8937-4f69-b7f0-25b2e4870336-istiod-ca-cert" pod "c3109f72-8937-4f69-b7f0-25b2e4870336" (UID: "c3109f72-8937-4f69-b7f0-25b2e4870336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:24:17.114753 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114620    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-certs" pod "c3109f72-8937-4f69-b7f0-25b2e4870336" (UID: "c3109f72-8937-4f69-b7f0-25b2e4870336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:17.114865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114828    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:17.114865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114853    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:17.114988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114868    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:17.114988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114881    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:17.114988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.114897    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3109f72-8937-4f69-b7f0-25b2e4870336-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:17.116027 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.115998    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-podinfo" pod "c3109f72-8937-4f69-b7f0-25b2e4870336" (UID: "c3109f72-8937-4f69-b7f0-25b2e4870336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:24:17.116178 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.116156    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-kube-api-access-nb7nb" pod "c3109f72-8937-4f69-b7f0-25b2e4870336" (UID: "c3109f72-8937-4f69-b7f0-25b2e4870336"). InnerVolumeSpecName "kube-api-access-nb7nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:24:17.116299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.116213    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-token" pod "c3109f72-8937-4f69-b7f0-25b2e4870336" (UID: "c3109f72-8937-4f69-b7f0-25b2e4870336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:24:17.116347 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.116336    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-envoy" pod "c3109f72-8937-4f69-b7f0-25b2e4870336" (UID: "c3109f72-8937-4f69-b7f0-25b2e4870336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:17.216333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.216244    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:17.216333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.216278    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:17.216333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.216289    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3109f72-8937-4f69-b7f0-25b2e4870336-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:17.216333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.216299    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb7nb\" (UniqueName: \"kubernetes.io/projected/c3109f72-8937-4f69-b7f0-25b2e4870336-kube-api-access-nb7nb\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:17.381680 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.381653    4882 generic.go:356] "Generic (PLEG): container finished" podID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerID="248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1" exitCode=0
Nov 01 04:24:17.381806 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.381737    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"
Nov 01 04:24:17.381806 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.381742    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" event={"ID":"c3109f72-8937-4f69-b7f0-25b2e4870336","Type":"ContainerDied","Data":"248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1"}
Nov 01 04:24:17.381952 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.381871    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs" event={"ID":"c3109f72-8937-4f69-b7f0-25b2e4870336","Type":"ContainerDied","Data":"0caff8734f68cbe2c379b8f05c79d6846c06bad229cab14605c4c72f85545930"}
Nov 01 04:24:17.381952 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.381908    4882 scope.go:122] "RemoveContainer" containerID="248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1"
Nov 01 04:24:17.384988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.384964    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j" event={"ID":"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef","Type":"ContainerStarted","Data":"81425bc4c2c6865eda85d70e3159248820f043ef64b2ff0f5874a0351e222814"}
Nov 01 04:24:17.392398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.392383    4882 scope.go:122] "RemoveContainer" containerID="80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa"
Nov 01 04:24:17.405881 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.405852    4882 scope.go:122] "RemoveContainer" containerID="248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1"
Nov 01 04:24:17.406775 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:24:17.406739    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1\": container with ID starting with 248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1 not found: ID does not exist" containerID="248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1"
Nov 01 04:24:17.406904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.406787    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1"} err="failed to get container status \"248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1\": rpc error: code = NotFound desc = could not find container \"248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1\": container with ID starting with 248715971ab437ea46bf99586bf45368821f72b3bea07072e9205029e4d153d1 not found: ID does not exist"
Nov 01 04:24:17.406904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.406812    4882 scope.go:122] "RemoveContainer" containerID="80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa"
Nov 01 04:24:17.407196 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:24:17.407175    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa\": container with ID starting with 80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa not found: ID does not exist" containerID="80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa"
Nov 01 04:24:17.407265 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.407208    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa"} err="failed to get container status \"80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa\": rpc error: code = NotFound desc = could not find container \"80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa\": container with ID starting with 80a4ed16256fdd3577864086b47bfbaf4a85927dab178f28e747f3e03f3879fa not found: ID does not exist"
Nov 01 04:24:17.536490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.536384    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"]
Nov 01 04:24:17.542391 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.542365    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ee0f8-openshift-default-5bbfd67bdf-qbzjs"]
Nov 01 04:24:17.601233 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.601200    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" path="/var/lib/kubelet/pods/c3109f72-8937-4f69-b7f0-25b2e4870336/volumes"
Nov 01 04:24:17.648224 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:17.648169    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j" podStartSLOduration=2.648154059 podStartE2EDuration="2.648154059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:24:17.645663966 +0000 UTC m=+2526.207696983" watchObservedRunningTime="2026-11-01 04:24:17.648154059 +0000 UTC m=+2526.210187076"
Nov 01 04:24:18.392650 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:18.392609    4882 generic.go:356] "Generic (PLEG): container finished" podID="ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef" containerID="7095fec4eb5a6c0a0888f76b0475e03bfc64ec0868935a5476d5dc85317bf62c" exitCode=0
Nov 01 04:24:18.393074 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:18.392690    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j" event={"ID":"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef","Type":"ContainerDied","Data":"7095fec4eb5a6c0a0888f76b0475e03bfc64ec0868935a5476d5dc85317bf62c"}
Nov 01 04:24:18.977981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:18.977956    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:24:19.042143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042060    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-credential-socket\") pod \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") "
Nov 01 04:24:19.042143 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042121    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-token\" (UniqueName: \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-token\") pod \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") "
Nov 01 04:24:19.042353 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042192    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istiod-ca-cert\") pod \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") "
Nov 01 04:24:19.042353 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042265    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-kube-api-access-vq99m\" (UniqueName: \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-kube-api-access-vq99m\") pod \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") "
Nov 01 04:24:19.042353 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042297    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-socket\") pod \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") "
Nov 01 04:24:19.042353 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042330    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-envoy\") pod \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") "
Nov 01 04:24:19.042353 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042334    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-credential-socket" pod "2984ced8-c4c5-44fa-b20c-6de34c0c891f" (UID: "2984ced8-c4c5-44fa-b20c-6de34c0c891f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:19.042654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042387    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-data\") pod \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") "
Nov 01 04:24:19.042654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042464    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-certs\") pod \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") "
Nov 01 04:24:19.042654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042531    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-podinfo\") pod \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\" (UID: \"2984ced8-c4c5-44fa-b20c-6de34c0c891f\") "
Nov 01 04:24:19.042654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042595    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-socket" pod "2984ced8-c4c5-44fa-b20c-6de34c0c891f" (UID: "2984ced8-c4c5-44fa-b20c-6de34c0c891f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:19.042889 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042733    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-data" pod "2984ced8-c4c5-44fa-b20c-6de34c0c891f" (UID: "2984ced8-c4c5-44fa-b20c-6de34c0c891f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:19.042889 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042758    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istiod-ca-cert" pod "2984ced8-c4c5-44fa-b20c-6de34c0c891f" (UID: "2984ced8-c4c5-44fa-b20c-6de34c0c891f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:24:19.042889 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.042818    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-certs" pod "2984ced8-c4c5-44fa-b20c-6de34c0c891f" (UID: "2984ced8-c4c5-44fa-b20c-6de34c0c891f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:19.043185 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.043160    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:19.043271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.043191    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:19.043271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.043208    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:19.043271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.043222    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:19.043271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.043236    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:19.044653 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.044627    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-token" pod "2984ced8-c4c5-44fa-b20c-6de34c0c891f" (UID: "2984ced8-c4c5-44fa-b20c-6de34c0c891f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:24:19.044744 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.044632    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-envoy" pod "2984ced8-c4c5-44fa-b20c-6de34c0c891f" (UID: "2984ced8-c4c5-44fa-b20c-6de34c0c891f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:24:19.044744 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.044735    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-podinfo" pod "2984ced8-c4c5-44fa-b20c-6de34c0c891f" (UID: "2984ced8-c4c5-44fa-b20c-6de34c0c891f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:24:19.044828 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.044744    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-kube-api-access-vq99m" pod "2984ced8-c4c5-44fa-b20c-6de34c0c891f" (UID: "2984ced8-c4c5-44fa-b20c-6de34c0c891f"). InnerVolumeSpecName "kube-api-access-vq99m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:24:19.144934 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.144898    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:19.144934 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.144930    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:19.144934 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.144939    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:19.145166 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.144948    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq99m\" (UniqueName: \"kubernetes.io/projected/2984ced8-c4c5-44fa-b20c-6de34c0c891f-kube-api-access-vq99m\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:19.401642 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.401552    4882 generic.go:356] "Generic (PLEG): container finished" podID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerID="ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7" exitCode=0
Nov 01 04:24:19.401642 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.401600    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"
Nov 01 04:24:19.401642 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.401630    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" event={"ID":"2984ced8-c4c5-44fa-b20c-6de34c0c891f","Type":"ContainerDied","Data":"ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7"}
Nov 01 04:24:19.402314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.401693    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf" event={"ID":"2984ced8-c4c5-44fa-b20c-6de34c0c891f","Type":"ContainerDied","Data":"5b39a8e2b816f4ccd13f444e16720680590745014c3cb4e0040010d43b4253e5"}
Nov 01 04:24:19.402314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.401718    4882 scope.go:122] "RemoveContainer" containerID="ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7"
Nov 01 04:24:19.412995 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.412970    4882 scope.go:122] "RemoveContainer" containerID="aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9"
Nov 01 04:24:19.423004 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.422985    4882 scope.go:122] "RemoveContainer" containerID="ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7"
Nov 01 04:24:19.423312 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:24:19.423292    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7\": container with ID starting with ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7 not found: ID does not exist" containerID="ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7"
Nov 01 04:24:19.423375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.423322    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7"} err="failed to get container status \"ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7\": rpc error: code = NotFound desc = could not find container \"ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7\": container with ID starting with ac68e3499056402ef13817877435958ba92f4ad03176da9d9dd6212ec904b2a7 not found: ID does not exist"
Nov 01 04:24:19.423375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.423343    4882 scope.go:122] "RemoveContainer" containerID="aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9"
Nov 01 04:24:19.423651 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:24:19.423632    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9\": container with ID starting with aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9 not found: ID does not exist" containerID="aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9"
Nov 01 04:24:19.423713 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.423661    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9"} err="failed to get container status \"aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9\": rpc error: code = NotFound desc = could not find container \"aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9\": container with ID starting with aab4f3425dd86ea2165406886a9990d56173535accaea8c2ae6de71704608fc9 not found: ID does not exist"
Nov 01 04:24:19.533907 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.533876    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:19.555155 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.555124    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"]
Nov 01 04:24:19.602664 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.602637    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4c2a66-openshift-default-54ffd7ff4f-ddnmf"]
Nov 01 04:24:19.651490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.651450    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-kube-api-access-h99jn\" (UniqueName: \"kubernetes.io/projected/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-kube-api-access-h99jn\") pod \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") "
Nov 01 04:24:19.651723 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.651559    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-openshift-ca\") pod \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") "
Nov 01 04:24:19.651723 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.651634    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-xks-ca\") pod \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\" (UID: \"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef\") "
Nov 01 04:24:19.651965 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.651905    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-openshift-ca" pod "ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef" (UID: "ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:24:19.652302 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.652281    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:19.653677 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.653654    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-xks-ca" pod "ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef" (UID: "ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:24:19.653770 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.653752    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-kube-api-access-h99jn" pod "ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef" (UID: "ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef"). InnerVolumeSpecName "kube-api-access-h99jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:24:19.753732 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.753702    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:19.753732 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:19.753731    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h99jn\" (UniqueName: \"kubernetes.io/projected/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef-kube-api-access-h99jn\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:20.410301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:20.410260    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j" event={"ID":"ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef","Type":"ContainerDied","Data":"81425bc4c2c6865eda85d70e3159248820f043ef64b2ff0f5874a0351e222814"}
Nov 01 04:24:20.410301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:20.410303    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"
Nov 01 04:24:20.410756 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:20.410322    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81425bc4c2c6865eda85d70e3159248820f043ef64b2ff0f5874a0351e222814"
Nov 01 04:24:21.599875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:21.599834    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" path="/var/lib/kubelet/pods/2984ced8-c4c5-44fa-b20c-6de34c0c891f/volumes"
Nov 01 04:24:26.665410 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:26.665371    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"]
Nov 01 04:24:26.763087 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:26.763057    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"]
Nov 01 04:24:28.722783 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:24:28.722747    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:24:29.314478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:29.314440    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"]
Nov 01 04:24:29.360129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:29.360093    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e7c9be-bg84x"]
Nov 01 04:24:29.600219 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:29.600145    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772e9836-4f5f-4f97-b356-3084cdb1c359" path="/var/lib/kubelet/pods/772e9836-4f5f-4f97-b356-3084cdb1c359/volumes"
Nov 01 04:24:31.666668 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:31.666605    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb" podUID="296c6127-1cf0-4538-9cd3-78a41fc4a9f2" containerName="payload-processing" containerID="cri-o://f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6" gracePeriod=25
Nov 01 04:24:31.764369 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:31.764316    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2" podUID="f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd" containerName="payload-pre-processing" containerID="cri-o://5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87" gracePeriod=25
Nov 01 04:24:31.931489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:31.931413    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:24:32.009851 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.009820    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-kube-api-access-xjbp5\" (UniqueName: \"kubernetes.io/projected/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-kube-api-access-xjbp5\") pod \"296c6127-1cf0-4538-9cd3-78a41fc4a9f2\" (UID: \"296c6127-1cf0-4538-9cd3-78a41fc4a9f2\") "
Nov 01 04:24:32.010029 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.009956    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-plugins-config-volume\") pod \"296c6127-1cf0-4538-9cd3-78a41fc4a9f2\" (UID: \"296c6127-1cf0-4538-9cd3-78a41fc4a9f2\") "
Nov 01 04:24:32.010316 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.010294    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-plugins-config-volume" pod "296c6127-1cf0-4538-9cd3-78a41fc4a9f2" (UID: "296c6127-1cf0-4538-9cd3-78a41fc4a9f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:24:32.011985 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.011957    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-kube-api-access-xjbp5" pod "296c6127-1cf0-4538-9cd3-78a41fc4a9f2" (UID: "296c6127-1cf0-4538-9cd3-78a41fc4a9f2"). InnerVolumeSpecName "kube-api-access-xjbp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:24:32.032999 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.032979    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:24:32.111335 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.111301    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-kube-api-access-2cj2h\" (UniqueName: \"kubernetes.io/projected/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-kube-api-access-2cj2h\") pod \"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd\" (UID: \"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd\") "
Nov 01 04:24:32.111558 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.111353    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-plugins-config-volume\") pod \"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd\" (UID: \"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd\") "
Nov 01 04:24:32.111661 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.111644    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjbp5\" (UniqueName: \"kubernetes.io/projected/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-kube-api-access-xjbp5\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:32.111732 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.111662    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/296c6127-1cf0-4538-9cd3-78a41fc4a9f2-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:32.111853 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.111831    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-plugins-config-volume" pod "f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd" (UID: "f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:24:32.113472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.113443    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-kube-api-access-2cj2h" pod "f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd" (UID: "f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd"). InnerVolumeSpecName "kube-api-access-2cj2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:24:32.212663 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.212630    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cj2h\" (UniqueName: \"kubernetes.io/projected/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-kube-api-access-2cj2h\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:32.212663 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.212658    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:32.496584 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.496499    4882 generic.go:356] "Generic (PLEG): container finished" podID="296c6127-1cf0-4538-9cd3-78a41fc4a9f2" containerID="f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6" exitCode=0
Nov 01 04:24:32.496584 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.496542    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"
Nov 01 04:24:32.496804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.496581    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb" event={"ID":"296c6127-1cf0-4538-9cd3-78a41fc4a9f2","Type":"ContainerDied","Data":"f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6"}
Nov 01 04:24:32.496804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.496638    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb" event={"ID":"296c6127-1cf0-4538-9cd3-78a41fc4a9f2","Type":"ContainerDied","Data":"4477e19cc55bfeb50a235432c78aac625ea25d478f027d3a574fa6ddb63ad736"}
Nov 01 04:24:32.496804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.496720    4882 scope.go:122] "RemoveContainer" containerID="f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6"
Nov 01 04:24:32.498075 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.498050    4882 generic.go:356] "Generic (PLEG): container finished" podID="f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd" containerID="5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87" exitCode=0
Nov 01 04:24:32.498197 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.498082    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2" event={"ID":"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd","Type":"ContainerDied","Data":"5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87"}
Nov 01 04:24:32.498197 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.498099    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"
Nov 01 04:24:32.498197 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.498111    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2" event={"ID":"f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd","Type":"ContainerDied","Data":"2e207bf920a23e83dbb01fffb87774c3499fccb545086ed2a80d8c9e0892e6b2"}
Nov 01 04:24:32.507474 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.507451    4882 scope.go:122] "RemoveContainer" containerID="f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6"
Nov 01 04:24:32.507782 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:24:32.507760    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6\": container with ID starting with f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6 not found: ID does not exist" containerID="f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6"
Nov 01 04:24:32.507879 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.507793    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6"} err="failed to get container status \"f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6\": rpc error: code = NotFound desc = could not find container \"f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6\": container with ID starting with f5a2b39b05dbef79a3be754b865b7f1d4fac97e486078fe61d4e16df27bcf6f6 not found: ID does not exist"
Nov 01 04:24:32.507879 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.507820    4882 scope.go:122] "RemoveContainer" containerID="5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87"
Nov 01 04:24:32.517712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.517690    4882 scope.go:122] "RemoveContainer" containerID="5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87"
Nov 01 04:24:32.518035 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:24:32.518016    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87\": container with ID starting with 5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87 not found: ID does not exist" containerID="5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87"
Nov 01 04:24:32.518119 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.518045    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87"} err="failed to get container status \"5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87\": rpc error: code = NotFound desc = could not find container \"5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87\": container with ID starting with 5244b5884487cea65fde100058657edd5e753eeb96b86a6a95dcba97f2cc4b87 not found: ID does not exist"
Nov 01 04:24:32.650093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.650060    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"]
Nov 01 04:24:32.696235 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.696207    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2afc0d-85d8cdf78c-sr5q2"]
Nov 01 04:24:32.797502 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.797403    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"]
Nov 01 04:24:32.800011 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:32.799986    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2afc0d-7577c5bc65-8jqqb"]
Nov 01 04:24:33.599834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:33.599803    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="296c6127-1cf0-4538-9cd3-78a41fc4a9f2" path="/var/lib/kubelet/pods/296c6127-1cf0-4538-9cd3-78a41fc4a9f2/volumes"
Nov 01 04:24:33.600230 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:33.600216    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd" path="/var/lib/kubelet/pods/f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd/volumes"
Nov 01 04:24:37.390637 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.390601    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"]
Nov 01 04:24:37.391314 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391290    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy"
Nov 01 04:24:37.391419 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391314    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy"
Nov 01 04:24:37.391419 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391340    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy"
Nov 01 04:24:37.391419 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391348    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy"
Nov 01 04:24:37.391419 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391370    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef" containerName="revoke-keys"
Nov 01 04:24:37.391419 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391377    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef" containerName="revoke-keys"
Nov 01 04:24:37.391419 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391399    4882 cpu_manager.go:414] "containerMap: removing container" podUID="296c6127-1cf0-4538-9cd3-78a41fc4a9f2" containerName="payload-processing"
Nov 01 04:24:37.391419 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391406    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296c6127-1cf0-4538-9cd3-78a41fc4a9f2" containerName="payload-processing"
Nov 01 04:24:37.391823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391449    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd" containerName="payload-pre-processing"
Nov 01 04:24:37.391823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391457    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd" containerName="payload-pre-processing"
Nov 01 04:24:37.391823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391468    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy"
Nov 01 04:24:37.391823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391476    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy"
Nov 01 04:24:37.391823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391612    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy"
Nov 01 04:24:37.391823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391629    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef" containerName="revoke-keys"
Nov 01 04:24:37.391823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391645    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="296c6127-1cf0-4538-9cd3-78a41fc4a9f2" containerName="payload-processing"
Nov 01 04:24:37.391823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391659    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d7d8f7-6ec4-4b8c-aa73-9f100fbe9ebd" containerName="payload-pre-processing"
Nov 01 04:24:37.391823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391668    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy"
Nov 01 04:24:37.391823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.391680    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3109f72-8937-4f69-b7f0-25b2e4870336" containerName="istio-proxy"
Nov 01 04:24:37.395275 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.395253    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:37.444978 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.444951    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:24:37.476466 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.476438    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:24:37.477033 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.477007    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:37.477151 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.477053    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:37.477259 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.477231    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvgvx\" (UniqueName: \"kubernetes.io/projected/5c43fc33-ed5e-4896-b109-48b051775787-kube-api-access-nvgvx\") pod \"maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:37.479311 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.479285    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"]
Nov 01 04:24:37.578291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.578253    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:37.578498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.578300    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:37.578498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.578390    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgvx\" (UniqueName: \"kubernetes.io/projected/5c43fc33-ed5e-4896-b109-48b051775787-kube-api-access-nvgvx\") pod \"maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:37.579054 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.579022    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:37.580848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.580827    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:37.636270 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.636238    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgvx\" (UniqueName: \"kubernetes.io/projected/5c43fc33-ed5e-4896-b109-48b051775787-kube-api-access-nvgvx\") pod \"maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:37.708394 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:37.708360    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:38.077707 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:38.077681    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"]
Nov 01 04:24:38.079805 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:24:38.079774    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c43fc33_ed5e_4896_b109_48b051775787.slice/crio-8fed0238af5c841787ba4bcb8ab9af8af609f5a6fd966b71d047cc7139fa9f47 WatchSource:0}: Error finding container 8fed0238af5c841787ba4bcb8ab9af8af609f5a6fd966b71d047cc7139fa9f47: Status 404 returned error can't find the container with id 8fed0238af5c841787ba4bcb8ab9af8af609f5a6fd966b71d047cc7139fa9f47
Nov 01 04:24:38.540051 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:38.540012    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22" event={"ID":"5c43fc33-ed5e-4896-b109-48b051775787","Type":"ContainerStarted","Data":"b47d770afcb216ee2becbbe04f6a4517f9c6802bb9bdda98b96ad4886f308ed8"}
Nov 01 04:24:38.540483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:38.540110    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22" event={"ID":"5c43fc33-ed5e-4896-b109-48b051775787","Type":"ContainerStarted","Data":"8fed0238af5c841787ba4bcb8ab9af8af609f5a6fd966b71d047cc7139fa9f47"}
Nov 01 04:24:39.548165 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:39.548125    4882 generic.go:356] "Generic (PLEG): container finished" podID="5c43fc33-ed5e-4896-b109-48b051775787" containerID="b47d770afcb216ee2becbbe04f6a4517f9c6802bb9bdda98b96ad4886f308ed8" exitCode=0
Nov 01 04:24:39.548591 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:39.548205    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22" event={"ID":"5c43fc33-ed5e-4896-b109-48b051775787","Type":"ContainerDied","Data":"b47d770afcb216ee2becbbe04f6a4517f9c6802bb9bdda98b96ad4886f308ed8"}
Nov 01 04:24:40.710024 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:40.709995    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:40.819811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:40.819721    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c43fc33-ed5e-4896-b109-48b051775787-kube-api-access-nvgvx\" (UniqueName: \"kubernetes.io/projected/5c43fc33-ed5e-4896-b109-48b051775787-kube-api-access-nvgvx\") pod \"5c43fc33-ed5e-4896-b109-48b051775787\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") "
Nov 01 04:24:40.819811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:40.819802    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-openshift-ca\") pod \"5c43fc33-ed5e-4896-b109-48b051775787\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") "
Nov 01 04:24:40.820096 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:40.819911    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-xks-ca\") pod \"5c43fc33-ed5e-4896-b109-48b051775787\" (UID: \"5c43fc33-ed5e-4896-b109-48b051775787\") "
Nov 01 04:24:40.820297 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:40.820160    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-openshift-ca" pod "5c43fc33-ed5e-4896-b109-48b051775787" (UID: "5c43fc33-ed5e-4896-b109-48b051775787"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:24:40.820483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:40.820461    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:40.822348 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:40.822324    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c43fc33-ed5e-4896-b109-48b051775787-kube-api-access-nvgvx" pod "5c43fc33-ed5e-4896-b109-48b051775787" (UID: "5c43fc33-ed5e-4896-b109-48b051775787"). InnerVolumeSpecName "kube-api-access-nvgvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:24:40.822610 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:40.822581    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-xks-ca" pod "5c43fc33-ed5e-4896-b109-48b051775787" (UID: "5c43fc33-ed5e-4896-b109-48b051775787"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:24:40.921468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:40.921404    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvgvx\" (UniqueName: \"kubernetes.io/projected/5c43fc33-ed5e-4896-b109-48b051775787-kube-api-access-nvgvx\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:40.921468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:40.921468    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c43fc33-ed5e-4896-b109-48b051775787-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:24:41.561852 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:41.561822    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"
Nov 01 04:24:41.562020 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:41.561847    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22" event={"ID":"5c43fc33-ed5e-4896-b109-48b051775787","Type":"ContainerDied","Data":"8fed0238af5c841787ba4bcb8ab9af8af609f5a6fd966b71d047cc7139fa9f47"}
Nov 01 04:24:41.562020 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:41.561892    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fed0238af5c841787ba4bcb8ab9af8af609f5a6fd966b71d047cc7139fa9f47"
Nov 01 04:24:57.081848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:57.081766    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"]
Nov 01 04:24:57.127553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:57.127524    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0882f4-xgl5f"]
Nov 01 04:24:57.599699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:24:57.599669    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab11570-7a8b-4d4d-b64b-0fbb9985c00a" path="/var/lib/kubelet/pods/bab11570-7a8b-4d4d-b64b-0fbb9985c00a/volumes"
Nov 01 04:24:58.722624 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:24:58.722589    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:25:03.790522 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:03.790492    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"]
Nov 01 04:25:03.791008 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:03.790738    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy" containerID="cri-o://cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719" gracePeriod=30
Nov 01 04:25:06.568437 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.568393    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"]
Nov 01 04:25:06.568945 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.568930    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy"
Nov 01 04:25:06.568992 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.568946    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy"
Nov 01 04:25:06.568992 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.568967    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5c43fc33-ed5e-4896-b109-48b051775787" containerName="revoke-keys"
Nov 01 04:25:06.568992 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.568973    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c43fc33-ed5e-4896-b109-48b051775787" containerName="revoke-keys"
Nov 01 04:25:06.569092 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.569086    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2984ced8-c4c5-44fa-b20c-6de34c0c891f" containerName="istio-proxy"
Nov 01 04:25:06.569126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.569103    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c43fc33-ed5e-4896-b109-48b051775787" containerName="revoke-keys"
Nov 01 04:25:06.572548 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.572518    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.621113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.621083    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-1cdd1d-openshift-default-dockercfg-r6b98\""
Nov 01 04:25:06.671794 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.671756    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"]
Nov 01 04:25:06.731806 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.731774    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5mh\" (UniqueName: \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-kube-api-access-qq5mh\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.732005 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.731815    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istiod-ca-cert\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.732005 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.731932    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-envoy\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.732185 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.732131    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-data\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.732233 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.732216    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-podinfo\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.732324 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.732311    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-socket\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.732371 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.732358    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-credential-socket\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.732409 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.732394    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-token\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.732486 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.732418    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-certs\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834034    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-credential-socket\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834072    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-token\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834325 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834137    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-certs\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834325 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834214    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5mh\" (UniqueName: \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-kube-api-access-qq5mh\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834325 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834242    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istiod-ca-cert\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834491 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834401    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-envoy\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834543 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834515    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-data\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834585 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834573    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-podinfo\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834669 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834652    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-socket\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834719 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834669    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-certs\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834718    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-credential-socket\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.834952 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.834921    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-data\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.835098 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.835042    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-socket\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.835098 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.835086    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istiod-ca-cert\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.836866 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.836846    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-envoy\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.837151 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.837132    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-podinfo\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.891325 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.891286    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-token\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:06.891535 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:06.891391    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5mh\" (UniqueName: \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-kube-api-access-qq5mh\") pod \"e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") " pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:07.186959 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:07.186925    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:07.615112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:07.615087    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"]
Nov 01 04:25:07.617150 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:25:07.617118    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9390f9be_7ca3_4bf4_9da5_65cdc7daba97.slice/crio-501c36fdf7922d917adcb3f5c8fbe6b485080623c416bb563997f0c2abe51c97 WatchSource:0}: Error finding container 501c36fdf7922d917adcb3f5c8fbe6b485080623c416bb563997f0c2abe51c97: Status 404 returned error can't find the container with id 501c36fdf7922d917adcb3f5c8fbe6b485080623c416bb563997f0c2abe51c97
Nov 01 04:25:07.619622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:07.619589    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:25:07.619739 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:07.619653    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:25:07.619739 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:07.619697    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:25:07.739609 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:07.739573    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" event={"ID":"9390f9be-7ca3-4bf4-9da5-65cdc7daba97","Type":"ContainerStarted","Data":"b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9"}
Nov 01 04:25:07.739791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:07.739638    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" event={"ID":"9390f9be-7ca3-4bf4-9da5-65cdc7daba97","Type":"ContainerStarted","Data":"501c36fdf7922d917adcb3f5c8fbe6b485080623c416bb563997f0c2abe51c97"}
Nov 01 04:25:07.846980 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:07.846940    4882 patch_prober.go:29] interesting pod/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:25:07.847137 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:07.847003    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:25:07.852715 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:07.852670    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" podStartSLOduration=1.852655402 podStartE2EDuration="1.852655402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:25:07.85101716 +0000 UTC m=+2576.413050178" watchObservedRunningTime="2026-11-01 04:25:07.852655402 +0000 UTC m=+2576.414688419"
Nov 01 04:25:08.187499 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:08.187463    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:08.192978 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:08.192950    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:08.745888 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:08.745858    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:08.747547 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:08.747520    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:25:09.055213 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.055190    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:25:09.164625 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.164592    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istiod-ca-cert\") pod \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") "
Nov 01 04:25:09.164811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.164647    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-podinfo\") pod \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") "
Nov 01 04:25:09.164811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.164680    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-kube-api-access-skbbs\" (UniqueName: \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-kube-api-access-skbbs\") pod \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") "
Nov 01 04:25:09.164811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.164702    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-certs\") pod \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") "
Nov 01 04:25:09.164811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.164746    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-data\") pod \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") "
Nov 01 04:25:09.164811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.164790    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-envoy\") pod \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") "
Nov 01 04:25:09.165072 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.164820    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-credential-socket\") pod \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") "
Nov 01 04:25:09.165072 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.164893    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-socket\") pod \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") "
Nov 01 04:25:09.165072 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.164974    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-token\" (UniqueName: \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-token\") pod \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\" (UID: \"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b\") "
Nov 01 04:25:09.165248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.165112    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-certs" pod "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" (UID: "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:09.165248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.165123    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istiod-ca-cert" pod "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" (UID: "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:25:09.165248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.165137    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-data" pod "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" (UID: "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:09.165411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.165348    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-credential-socket" pod "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" (UID: "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:09.165411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.165360    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-socket" pod "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" (UID: "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:09.165636 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.165618    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:09.165715 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.165640    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:09.165715 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.165658    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:09.165715 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.165667    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:09.165715 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.165676    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:09.167103 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.167079    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-podinfo" pod "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" (UID: "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:25:09.167284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.167214    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-kube-api-access-skbbs" pod "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" (UID: "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b"). InnerVolumeSpecName "kube-api-access-skbbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:25:09.167469 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.167416    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-envoy" pod "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" (UID: "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:09.167539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.167477    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-token" pod "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" (UID: "bd5256b4-e61e-4ac2-82b5-9f28bc217d0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:25:09.274737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.274703    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:09.274737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.274733    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skbbs\" (UniqueName: \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-kube-api-access-skbbs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:09.274737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.274745    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:09.274983 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.274753    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:09.755558 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.755524    4882 generic.go:356] "Generic (PLEG): container finished" podID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerID="cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719" exitCode=0
Nov 01 04:25:09.756037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.755593    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"
Nov 01 04:25:09.756037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.755602    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" event={"ID":"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b","Type":"ContainerDied","Data":"cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719"}
Nov 01 04:25:09.756037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.755671    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l" event={"ID":"bd5256b4-e61e-4ac2-82b5-9f28bc217d0b","Type":"ContainerDied","Data":"ff74d15f2d396add0bb8d660e2cb3147ba752a5d2d3ca7ce6d7fdd81c707e289"}
Nov 01 04:25:09.756037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.755701    4882 scope.go:122] "RemoveContainer" containerID="cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719"
Nov 01 04:25:09.768796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.768769    4882 scope.go:122] "RemoveContainer" containerID="28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181"
Nov 01 04:25:09.778252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.778235    4882 scope.go:122] "RemoveContainer" containerID="cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719"
Nov 01 04:25:09.778557 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:25:09.778534    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719\": container with ID starting with cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719 not found: ID does not exist" containerID="cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719"
Nov 01 04:25:09.778628 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.778564    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719"} err="failed to get container status \"cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719\": rpc error: code = NotFound desc = could not find container \"cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719\": container with ID starting with cedcfb2520cfff1a94418907e1cc9c839ed71b7017b9a9a63c16489448558719 not found: ID does not exist"
Nov 01 04:25:09.778628 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.778586    4882 scope.go:122] "RemoveContainer" containerID="28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181"
Nov 01 04:25:09.778904 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:25:09.778867    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181\": container with ID starting with 28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181 not found: ID does not exist" containerID="28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181"
Nov 01 04:25:09.778953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.778910    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181"} err="failed to get container status \"28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181\": rpc error: code = NotFound desc = could not find container \"28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181\": container with ID starting with 28a4612814f62548df8165e2967386d147d0d491ca51b61cd3d0eaed42753181 not found: ID does not exist"
Nov 01 04:25:09.909307 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.909229    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"]
Nov 01 04:25:09.955864 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:09.955822    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0882f4-openshift-default-86d8dcd84f-bhd8l"]
Nov 01 04:25:11.600706 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:11.600678    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" path="/var/lib/kubelet/pods/bd5256b4-e61e-4ac2-82b5-9f28bc217d0b/volumes"
Nov 01 04:25:19.340490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:19.340451    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"]
Nov 01 04:25:19.386527 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:19.386497    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2afc0d-8ql8j"]
Nov 01 04:25:19.599331 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:19.599254    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef" path="/var/lib/kubelet/pods/ed2d747f-37e6-4d64-ba8e-6b5bc33ac9ef/volumes"
Nov 01 04:25:26.054474 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:26.054417    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"]
Nov 01 04:25:26.054984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:26.054647    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy" containerID="cri-o://3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255" gracePeriod=30
Nov 01 04:25:27.176059 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.176026    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"]
Nov 01 04:25:27.176581 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.176565    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy"
Nov 01 04:25:27.176635 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.176582    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy"
Nov 01 04:25:27.176635 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.176612    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy"
Nov 01 04:25:27.176635 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.176618    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy"
Nov 01 04:25:27.176735 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.176705    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy"
Nov 01 04:25:27.181534 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.181513    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:27.230299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.230263    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:25:27.276223 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.276195    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:25:27.279423 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.279390    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"]
Nov 01 04:25:27.293191 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.293162    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cc2h\" (UniqueName: \"kubernetes.io/projected/8a4059da-8449-498f-a379-a5e407d52027-kube-api-access-7cc2h\") pod \"maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:27.293332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.293207    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4059da-8449-498f-a379-a5e407d52027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:27.293332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.293236    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4059da-8449-498f-a379-a5e407d52027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:27.358671 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.358644    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:25:27.358804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.358694    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:25:27.394105 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.394077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cc2h\" (UniqueName: \"kubernetes.io/projected/8a4059da-8449-498f-a379-a5e407d52027-kube-api-access-7cc2h\") pod \"maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:27.394247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.394119    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4059da-8449-498f-a379-a5e407d52027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:27.394247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.394140    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4059da-8449-498f-a379-a5e407d52027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:27.394886 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.394867    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4059da-8449-498f-a379-a5e407d52027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:27.396478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.396456    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4059da-8449-498f-a379-a5e407d52027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:27.451052 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.451020    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cc2h\" (UniqueName: \"kubernetes.io/projected/8a4059da-8449-498f-a379-a5e407d52027-kube-api-access-7cc2h\") pod \"maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:27.494544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.494520    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:27.864501 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:27.864474    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"]
Nov 01 04:25:27.866447 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:25:27.866396    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4059da_8449_498f_a379_a5e407d52027.slice/crio-5f2ca7c776f045ada0f5ea847b5a0c414fde772022f45ccc616db058d9e639d4 WatchSource:0}: Error finding container 5f2ca7c776f045ada0f5ea847b5a0c414fde772022f45ccc616db058d9e639d4: Status 404 returned error can't find the container with id 5f2ca7c776f045ada0f5ea847b5a0c414fde772022f45ccc616db058d9e639d4
Nov 01 04:25:28.723226 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:25:28.723191    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:25:28.879004 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:28.878968    4882 generic.go:356] "Generic (PLEG): container finished" podID="8a4059da-8449-498f-a379-a5e407d52027" containerID="a18cd009376ef9ce507bf6a3be1523f34fca950ae2bfcf03ac6119bb5215bdb5" exitCode=0
Nov 01 04:25:28.879174 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:28.879052    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c" event={"ID":"8a4059da-8449-498f-a379-a5e407d52027","Type":"ContainerDied","Data":"a18cd009376ef9ce507bf6a3be1523f34fca950ae2bfcf03ac6119bb5215bdb5"}
Nov 01 04:25:28.879174 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:28.879104    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c" event={"ID":"8a4059da-8449-498f-a379-a5e407d52027","Type":"ContainerStarted","Data":"5f2ca7c776f045ada0f5ea847b5a0c414fde772022f45ccc616db058d9e639d4"}
Nov 01 04:25:30.024130 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.024105    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:30.125906 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.125879    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a4059da-8449-498f-a379-a5e407d52027-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4059da-8449-498f-a379-a5e407d52027-maas-api-xks-ca\") pod \"8a4059da-8449-498f-a379-a5e407d52027\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") "
Nov 01 04:25:30.126088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.125991    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a4059da-8449-498f-a379-a5e407d52027-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4059da-8449-498f-a379-a5e407d52027-maas-api-openshift-ca\") pod \"8a4059da-8449-498f-a379-a5e407d52027\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") "
Nov 01 04:25:30.126088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.126011    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4059da-8449-498f-a379-a5e407d52027-kube-api-access-7cc2h\" (UniqueName: \"kubernetes.io/projected/8a4059da-8449-498f-a379-a5e407d52027-kube-api-access-7cc2h\") pod \"8a4059da-8449-498f-a379-a5e407d52027\" (UID: \"8a4059da-8449-498f-a379-a5e407d52027\") "
Nov 01 04:25:30.126361 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.126337    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4059da-8449-498f-a379-a5e407d52027-maas-api-openshift-ca" pod "8a4059da-8449-498f-a379-a5e407d52027" (UID: "8a4059da-8449-498f-a379-a5e407d52027"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:25:30.128129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.128110    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a4059da-8449-498f-a379-a5e407d52027-maas-api-xks-ca" pod "8a4059da-8449-498f-a379-a5e407d52027" (UID: "8a4059da-8449-498f-a379-a5e407d52027"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:25:30.128239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.128223    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4059da-8449-498f-a379-a5e407d52027-kube-api-access-7cc2h" pod "8a4059da-8449-498f-a379-a5e407d52027" (UID: "8a4059da-8449-498f-a379-a5e407d52027"). InnerVolumeSpecName "kube-api-access-7cc2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:25:30.227016 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.226988    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a4059da-8449-498f-a379-a5e407d52027-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:30.227016 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.227014    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a4059da-8449-498f-a379-a5e407d52027-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:30.227212 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.227025    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cc2h\" (UniqueName: \"kubernetes.io/projected/8a4059da-8449-498f-a379-a5e407d52027-kube-api-access-7cc2h\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:30.892672 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.892643    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"
Nov 01 04:25:30.892672 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.892663    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c" event={"ID":"8a4059da-8449-498f-a379-a5e407d52027","Type":"ContainerDied","Data":"5f2ca7c776f045ada0f5ea847b5a0c414fde772022f45ccc616db058d9e639d4"}
Nov 01 04:25:30.892936 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:30.892703    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2ca7c776f045ada0f5ea847b5a0c414fde772022f45ccc616db058d9e639d4"
Nov 01 04:25:31.328474 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.328445    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:25:31.440196 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440157    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-credential-socket\") pod \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") "
Nov 01 04:25:31.440377 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440214    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-envoy\") pod \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") "
Nov 01 04:25:31.440377 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440350    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-kube-api-access-m9c7p\" (UniqueName: \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-kube-api-access-m9c7p\") pod \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") "
Nov 01 04:25:31.440500 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440411    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-socket\") pod \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") "
Nov 01 04:25:31.440559 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440536    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-credential-socket" pod "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" (UID: "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:31.440623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440561    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-certs\") pod \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") "
Nov 01 04:25:31.440623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440591    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-token\" (UniqueName: \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-token\") pod \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") "
Nov 01 04:25:31.440623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440610    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istiod-ca-cert\") pod \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") "
Nov 01 04:25:31.440791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440653    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-data\") pod \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") "
Nov 01 04:25:31.440791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440683    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-podinfo\") pod \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\" (UID: \"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b\") "
Nov 01 04:25:31.440791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440746    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-socket" pod "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" (UID: "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:31.440791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440763    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-certs" pod "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" (UID: "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:31.441012 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440924    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-data" pod "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" (UID: "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:31.441079 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.440999    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istiod-ca-cert" pod "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" (UID: "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:25:31.441210 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.441189    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:31.441287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.441217    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:31.441287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.441232    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:31.441287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.441246    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:31.441287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.441258    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:31.442831 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.442806    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-token" pod "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" (UID: "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:25:31.442999 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.442878    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-envoy" pod "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" (UID: "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:31.442999 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.442906    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-kube-api-access-m9c7p" pod "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" (UID: "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b"). InnerVolumeSpecName "kube-api-access-m9c7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:25:31.442999 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.442940    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-podinfo" pod "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" (UID: "f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:25:31.542561 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.542529    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:31.542561 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.542554    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:31.542561 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.542563    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:31.542871 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.542573    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9c7p\" (UniqueName: \"kubernetes.io/projected/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b-kube-api-access-m9c7p\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:31.582816 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.582783    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"]
Nov 01 04:25:31.600770 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.600742    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"]
Nov 01 04:25:31.900164 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.900074    4882 generic.go:356] "Generic (PLEG): container finished" podID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerID="3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255" exitCode=0
Nov 01 04:25:31.900164 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.900141    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"
Nov 01 04:25:31.900410 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.900157    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" event={"ID":"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b","Type":"ContainerDied","Data":"3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255"}
Nov 01 04:25:31.900410 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.900224    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84" event={"ID":"f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b","Type":"ContainerDied","Data":"228578076ad23a7fa95efaac0d20ddce8f02a5d6977fdaee1cae2592395d2314"}
Nov 01 04:25:31.900410 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.900255    4882 scope.go:122] "RemoveContainer" containerID="3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255"
Nov 01 04:25:31.910661 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.910641    4882 scope.go:122] "RemoveContainer" containerID="3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee"
Nov 01 04:25:31.919493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.919477    4882 scope.go:122] "RemoveContainer" containerID="3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255"
Nov 01 04:25:31.919759 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:25:31.919734    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255\": container with ID starting with 3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255 not found: ID does not exist" containerID="3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255"
Nov 01 04:25:31.919865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.919766    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255"} err="failed to get container status \"3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255\": rpc error: code = NotFound desc = could not find container \"3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255\": container with ID starting with 3c93bce6cc64aa8b4ce0358b62bca628cc76c1c8b344bc0339cdff48da906255 not found: ID does not exist"
Nov 01 04:25:31.919865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.919792    4882 scope.go:122] "RemoveContainer" containerID="3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee"
Nov 01 04:25:31.920048 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:25:31.920029    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee\": container with ID starting with 3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee not found: ID does not exist" containerID="3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee"
Nov 01 04:25:31.920094 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:31.920058    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee"} err="failed to get container status \"3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee\": rpc error: code = NotFound desc = could not find container \"3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee\": container with ID starting with 3ae6e9f471ca263f9affaaf27c5791297ec17d4bd0316986cd804a6975503bee not found: ID does not exist"
Nov 01 04:25:32.054497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:32.054459    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"]
Nov 01 04:25:32.058770 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:32.058738    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-2afc0d-openshift-default-75cd6c49f7-hmr84"]
Nov 01 04:25:33.600216 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:33.600182    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" path="/var/lib/kubelet/pods/f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b/volumes"
Nov 01 04:25:41.290741 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:41.290705    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"]
Nov 01 04:25:41.335673 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:41.335587    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-862ad1-mvq22"]
Nov 01 04:25:41.601624 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:41.601547    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c43fc33-ed5e-4896-b109-48b051775787" path="/var/lib/kubelet/pods/5c43fc33-ed5e-4896-b109-48b051775787/volumes"
Nov 01 04:25:46.013078 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.013043    4882 generic.go:356] "Generic (PLEG): container finished" podID="71b18693-924b-492c-9db8-29a52ef1bb12" containerID="721cb0d78fb595f35fdcbda10a830219c40705e52ec9d66ab3afb43afe565a73" exitCode=0
Nov 01 04:25:46.013586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.013120    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" event={"ID":"71b18693-924b-492c-9db8-29a52ef1bb12","Type":"ContainerDied","Data":"721cb0d78fb595f35fdcbda10a830219c40705e52ec9d66ab3afb43afe565a73"}
Nov 01 04:25:46.013586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.013551    4882 scope.go:122] "RemoveContainer" containerID="721cb0d78fb595f35fdcbda10a830219c40705e52ec9d66ab3afb43afe565a73"
Nov 01 04:25:46.015030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.015008    4882 generic.go:356] "Generic (PLEG): container finished" podID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerID="a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714" exitCode=0
Nov 01 04:25:46.015161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.015070    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:25:46.015161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.015091    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" event={"ID":"d8ca8add-0434-46a9-a055-0306697ac2d7","Type":"ContainerDied","Data":"a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714"}
Nov 01 04:25:46.015161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.015139    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:25:46.015360 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.015177    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:25:46.015516 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.015494    4882 scope.go:122] "RemoveContainer" containerID="a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714"
Nov 01 04:25:46.016830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.016802    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:25:46.016937 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.016862    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:25:46.016937 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.016894    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:25:46.372820 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.372741    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:25:46.607463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.604940    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerName="main" containerID="cri-o://11487d72df8cc4ab17a47e5d9d3cd0a28cbf372e18d9eb92fdd8878a900f21f2" gracePeriod=45
Nov 01 04:25:46.623280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.623193    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" containerID="cri-o://58fc46b431fa03455d050ba6e6831976682041dcd02581e5ca7ce65a1dbaee43" gracePeriod=45
Nov 01 04:25:46.773454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:46.769833    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused"
Nov 01 04:25:47.046012 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.045959    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" event={"ID":"a6635a62-77fe-4864-9430-c372b90d2e85","Type":"ContainerDied","Data":"58fc46b431fa03455d050ba6e6831976682041dcd02581e5ca7ce65a1dbaee43"}
Nov 01 04:25:47.046357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.046337    4882 generic.go:356] "Generic (PLEG): container finished" podID="a6635a62-77fe-4864-9430-c372b90d2e85" containerID="58fc46b431fa03455d050ba6e6831976682041dcd02581e5ca7ce65a1dbaee43" exitCode=0
Nov 01 04:25:47.046519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.046463    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2" event={"ID":"a6635a62-77fe-4864-9430-c372b90d2e85","Type":"ContainerDied","Data":"8460fec174d06d365abb399199227ad77f5c1d3c0a0fea79163597ee057612c5"}
Nov 01 04:25:47.046519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.046486    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8460fec174d06d365abb399199227ad77f5c1d3c0a0fea79163597ee057612c5"
Nov 01 04:25:47.049448 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.048284    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 04:25:47.049448 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.048817    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" event={"ID":"71b18693-924b-492c-9db8-29a52ef1bb12","Type":"ContainerStarted","Data":"97f458c9d4d40862cde63fd868ef69dfd4749c392aac59bdeb509ea8741439f5"}
Nov 01 04:25:47.052234 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.051710    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" event={"ID":"d8ca8add-0434-46a9-a055-0306697ac2d7","Type":"ContainerStarted","Data":"3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e"}
Nov 01 04:25:47.053683 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.053663    4882 generic.go:356] "Generic (PLEG): container finished" podID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerID="11487d72df8cc4ab17a47e5d9d3cd0a28cbf372e18d9eb92fdd8878a900f21f2" exitCode=0
Nov 01 04:25:47.053777 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.053739    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" event={"ID":"97c01b72-75a0-4153-819c-0f37c6b7d427","Type":"ContainerDied","Data":"11487d72df8cc4ab17a47e5d9d3cd0a28cbf372e18d9eb92fdd8878a900f21f2"}
Nov 01 04:25:47.057335 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.057321    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:25:47.117105 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.117070    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:25:47.117309 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.117211    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 04:25:47.117309 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.117268    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.175840    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c01b72-75a0-4153-819c-0f37c6b7d427-kube-api-access-l9md9\" (UniqueName: \"kubernetes.io/projected/97c01b72-75a0-4153-819c-0f37c6b7d427-kube-api-access-l9md9\") pod \"97c01b72-75a0-4153-819c-0f37c6b7d427\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") "
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.175933    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-tmp-dir\") pod \"97c01b72-75a0-4153-819c-0f37c6b7d427\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") "
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.176000    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c01b72-75a0-4153-819c-0f37c6b7d427-tls-certs\" (UniqueName: \"kubernetes.io/secret/97c01b72-75a0-4153-819c-0f37c6b7d427-tls-certs\") pod \"97c01b72-75a0-4153-819c-0f37c6b7d427\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") "
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.176043    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-dshm\") pod \"a6635a62-77fe-4864-9430-c372b90d2e85\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") "
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.176106    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-home\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-home\") pod \"97c01b72-75a0-4153-819c-0f37c6b7d427\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") "
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.176140    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-model-cache\") pod \"97c01b72-75a0-4153-819c-0f37c6b7d427\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") "
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.176192    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-home\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-home\") pod \"a6635a62-77fe-4864-9430-c372b90d2e85\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") "
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.176221    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-model-cache\") pod \"a6635a62-77fe-4864-9430-c372b90d2e85\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") "
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.176274    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-tmp-dir\") pod \"a6635a62-77fe-4864-9430-c372b90d2e85\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") "
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.176306    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6635a62-77fe-4864-9430-c372b90d2e85-kube-api-access-pcnwb\" (UniqueName: \"kubernetes.io/projected/a6635a62-77fe-4864-9430-c372b90d2e85-kube-api-access-pcnwb\") pod \"a6635a62-77fe-4864-9430-c372b90d2e85\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") "
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.176341    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-dshm\") pod \"97c01b72-75a0-4153-819c-0f37c6b7d427\" (UID: \"97c01b72-75a0-4153-819c-0f37c6b7d427\") "
Nov 01 04:25:47.178514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.176460    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6635a62-77fe-4864-9430-c372b90d2e85-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6635a62-77fe-4864-9430-c372b90d2e85-tls-certs\") pod \"a6635a62-77fe-4864-9430-c372b90d2e85\" (UID: \"a6635a62-77fe-4864-9430-c372b90d2e85\") "
Nov 01 04:25:47.202476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.189672    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-home" pod "97c01b72-75a0-4153-819c-0f37c6b7d427" (UID: "97c01b72-75a0-4153-819c-0f37c6b7d427"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:47.202476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.189851    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-tmp-dir" pod "97c01b72-75a0-4153-819c-0f37c6b7d427" (UID: "97c01b72-75a0-4153-819c-0f37c6b7d427"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:47.202476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.190998    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-home" pod "a6635a62-77fe-4864-9430-c372b90d2e85" (UID: "a6635a62-77fe-4864-9430-c372b90d2e85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:47.202476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.191199    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-model-cache" pod "97c01b72-75a0-4153-819c-0f37c6b7d427" (UID: "97c01b72-75a0-4153-819c-0f37c6b7d427"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:47.204038 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.203100    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-tmp-dir" pod "a6635a62-77fe-4864-9430-c372b90d2e85" (UID: "a6635a62-77fe-4864-9430-c372b90d2e85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:47.220633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.220567    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-model-cache" pod "a6635a62-77fe-4864-9430-c372b90d2e85" (UID: "a6635a62-77fe-4864-9430-c372b90d2e85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:47.222489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.222401    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c01b72-75a0-4153-819c-0f37c6b7d427-tls-certs" pod "97c01b72-75a0-4153-819c-0f37c6b7d427" (UID: "97c01b72-75a0-4153-819c-0f37c6b7d427"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:25:47.223572 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.223543    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-dshm" pod "a6635a62-77fe-4864-9430-c372b90d2e85" (UID: "a6635a62-77fe-4864-9430-c372b90d2e85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:47.223669 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.223638    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-dshm" pod "97c01b72-75a0-4153-819c-0f37c6b7d427" (UID: "97c01b72-75a0-4153-819c-0f37c6b7d427"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:47.223822 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.223794    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6635a62-77fe-4864-9430-c372b90d2e85-kube-api-access-pcnwb" pod "a6635a62-77fe-4864-9430-c372b90d2e85" (UID: "a6635a62-77fe-4864-9430-c372b90d2e85"). InnerVolumeSpecName "kube-api-access-pcnwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:25:47.223982 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.223960    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c01b72-75a0-4153-819c-0f37c6b7d427-kube-api-access-l9md9" pod "97c01b72-75a0-4153-819c-0f37c6b7d427" (UID: "97c01b72-75a0-4153-819c-0f37c6b7d427"). InnerVolumeSpecName "kube-api-access-l9md9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:25:47.224582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.224561    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6635a62-77fe-4864-9430-c372b90d2e85-tls-certs" pod "a6635a62-77fe-4864-9430-c372b90d2e85" (UID: "a6635a62-77fe-4864-9430-c372b90d2e85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310524    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-tmp-dir\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310578    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c01b72-75a0-4153-819c-0f37c6b7d427-tls-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310592    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-dshm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310606    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-home\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310620    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-model-cache\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310633    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-home\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310646    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-model-cache\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310660    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6635a62-77fe-4864-9430-c372b90d2e85-tmp-dir\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310675    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcnwb\" (UniqueName: \"kubernetes.io/projected/a6635a62-77fe-4864-9430-c372b90d2e85-kube-api-access-pcnwb\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310690    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97c01b72-75a0-4153-819c-0f37c6b7d427-dshm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310703    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6635a62-77fe-4864-9430-c372b90d2e85-tls-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.312454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.310716    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9md9\" (UniqueName: \"kubernetes.io/projected/97c01b72-75a0-4153-819c-0f37c6b7d427-kube-api-access-l9md9\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:47.408452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.405191    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:25:47.408452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.405329    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 04:25:47.408452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:47.405389    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 04:25:48.061135 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.061047    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"
Nov 01 04:25:48.061135 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.061074    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9" event={"ID":"97c01b72-75a0-4153-819c-0f37c6b7d427","Type":"ContainerDied","Data":"3f3a77ce0d184a156e5d4779293f36da9e42e2ecb9ec11afb3ae4213339b1e5d"}
Nov 01 04:25:48.061135 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.061120    4882 scope.go:122] "RemoveContainer" containerID="11487d72df8cc4ab17a47e5d9d3cd0a28cbf372e18d9eb92fdd8878a900f21f2"
Nov 01 04:25:48.061723 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.061690    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"
Nov 01 04:25:48.094614 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.088812    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 04:25:48.094614 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.088869    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 04:25:48.216668 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.216620    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"]
Nov 01 04:25:48.245933 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.245900    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/simulator-w3-f73b9f-kserve-5d96b65797-smhf2"]
Nov 01 04:25:48.333681 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.333599    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"]
Nov 01 04:25:48.337712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.337687    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-w3-f73b9f-kserve-67cc5475-7lsd9"]
Nov 01 04:25:48.428547 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.418230    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"]
Nov 01 04:25:48.428547 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.418575    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy" containerID="cri-o://3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827" gracePeriod=30
Nov 01 04:25:48.428547 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.427931    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 04:25:48.428547 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:48.427980    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 04:25:49.099450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:49.098827    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 04:25:49.099450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:49.098889    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 04:25:49.410512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:49.404241    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 04:25:49.410512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:49.404302    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 04:25:49.623631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:49.623602    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" path="/var/lib/kubelet/pods/97c01b72-75a0-4153-819c-0f37c6b7d427/volumes"
Nov 01 04:25:49.624079 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:49.624065    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" path="/var/lib/kubelet/pods/a6635a62-77fe-4864-9430-c372b90d2e85/volumes"
Nov 01 04:25:50.087452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.086858    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 04:25:50.087452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.086919    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 04:25:50.408510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.407820    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 04:25:50.408510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.407887    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 04:25:50.745098 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.745066    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"]
Nov 01 04:25:50.745773 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.745754    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy"
Nov 01 04:25:50.745903 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.745892    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy"
Nov 01 04:25:50.745973 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.745966    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8a4059da-8449-498f-a379-a5e407d52027" containerName="revoke-keys"
Nov 01 04:25:50.746027 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.746020    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4059da-8449-498f-a379-a5e407d52027" containerName="revoke-keys"
Nov 01 04:25:50.746093 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.746087    4882 cpu_manager.go:414] "containerMap: removing container" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerName="main"
Nov 01 04:25:50.746145 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.746138    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerName="main"
Nov 01 04:25:50.746213 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.746207    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main"
Nov 01 04:25:50.746261 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.746255    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main"
Nov 01 04:25:50.746423 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.746412    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy"
Nov 01 04:25:50.746510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.746502    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4059da-8449-498f-a379-a5e407d52027" containerName="revoke-keys"
Nov 01 04:25:50.746561 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.746555    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c01b72-75a0-4153-819c-0f37c6b7d427" containerName="main"
Nov 01 04:25:50.746617 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.746611    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy"
Nov 01 04:25:50.746668 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.746662    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6635a62-77fe-4864-9430-c372b90d2e85" containerName="main"
Nov 01 04:25:50.746719 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.746712    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5256b4-e61e-4ac2-82b5-9f28bc217d0b" containerName="istio-proxy"
Nov 01 04:25:50.751277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.751260    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:50.813069 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.813031    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"]
Nov 01 04:25:50.852294 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.852255    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:25:50.852482 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.852347    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:25:50.858858 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.858822    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmk76\" (UniqueName: \"kubernetes.io/projected/43a9a36c-fe33-4582-959f-02b39fce15fb-kube-api-access-gmk76\") pod \"maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:50.858986 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.858944    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:50.859032 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.858990    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:50.959736 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.959696    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmk76\" (UniqueName: \"kubernetes.io/projected/43a9a36c-fe33-4582-959f-02b39fce15fb-kube-api-access-gmk76\") pod \"maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:50.959942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.959781    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:50.959942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.959830    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:50.960623 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.960592    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:50.981073 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:50.974841    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:51.019406 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:51.019324    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmk76\" (UniqueName: \"kubernetes.io/projected/43a9a36c-fe33-4582-959f-02b39fce15fb-kube-api-access-gmk76\") pod \"maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:51.069452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:51.065577    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:51.112544 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:51.112517    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 04:25:51.112829 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:51.112808    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 04:25:51.412510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:51.411954    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 04:25:51.412510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:51.412010    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 04:25:51.512873 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:51.510231    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"]
Nov 01 04:25:52.088992 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:52.088887    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 01 04:25:52.089278 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:52.089234    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 01 04:25:52.139065 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:52.139013    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x" event={"ID":"43a9a36c-fe33-4582-959f-02b39fce15fb","Type":"ContainerStarted","Data":"1587910ac3e3a835a0ab016dccda508671cb847cd5489d5e1165afbb227133f0"}
Nov 01 04:25:52.139065 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:52.139070    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x" event={"ID":"43a9a36c-fe33-4582-959f-02b39fce15fb","Type":"ContainerStarted","Data":"d798604186462ddb21c3ebbacda7a8ca5636e37f6bcc23bc5ffa91d4b89cdc7f"}
Nov 01 04:25:52.247103 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:52.247043    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x" podStartSLOduration=2.247028133 podStartE2EDuration="2.247028133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:25:52.246073048 +0000 UTC m=+2620.808106067" watchObservedRunningTime="2026-11-01 04:25:52.247028133 +0000 UTC m=+2620.809061149"
Nov 01 04:25:52.406395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:52.406316    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:25:52.406640 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:52.406621    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:25:52.407168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:52.407152    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:25:53.090190 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.090164    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:25:53.090677 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.090542    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:25:53.091133 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.091118    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:25:53.155624 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.155588    4882 generic.go:356] "Generic (PLEG): container finished" podID="43a9a36c-fe33-4582-959f-02b39fce15fb" containerID="1587910ac3e3a835a0ab016dccda508671cb847cd5489d5e1165afbb227133f0" exitCode=0
Nov 01 04:25:53.155779 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.155662    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x" event={"ID":"43a9a36c-fe33-4582-959f-02b39fce15fb","Type":"ContainerDied","Data":"1587910ac3e3a835a0ab016dccda508671cb847cd5489d5e1165afbb227133f0"}
Nov 01 04:25:53.687600 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.687571    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:25:53.836749 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.836664    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-data\") pod \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") "
Nov 01 04:25:53.836895 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.836768    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-podinfo\") pod \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") "
Nov 01 04:25:53.836895 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.836853    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-certs\") pod \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") "
Nov 01 04:25:53.837025 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.836907    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istiod-ca-cert\") pod \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") "
Nov 01 04:25:53.837025 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.836966    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-token\" (UniqueName: \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-token\") pod \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") "
Nov 01 04:25:53.837025 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.836988    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-data" pod "81ce7003-30a0-4d13-a9be-cf28b38a1a7d" (UID: "81ce7003-30a0-4d13-a9be-cf28b38a1a7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:53.837025 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.836997    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-kube-api-access-pdz8f\" (UniqueName: \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-kube-api-access-pdz8f\") pod \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") "
Nov 01 04:25:53.837246 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.837077    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-envoy\") pod \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") "
Nov 01 04:25:53.837246 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.837203    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-credential-socket\") pod \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") "
Nov 01 04:25:53.837570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.837510    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-credential-socket" pod "81ce7003-30a0-4d13-a9be-cf28b38a1a7d" (UID: "81ce7003-30a0-4d13-a9be-cf28b38a1a7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:53.837570 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.837556    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-certs" pod "81ce7003-30a0-4d13-a9be-cf28b38a1a7d" (UID: "81ce7003-30a0-4d13-a9be-cf28b38a1a7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:53.837790 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.837669    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istiod-ca-cert" pod "81ce7003-30a0-4d13-a9be-cf28b38a1a7d" (UID: "81ce7003-30a0-4d13-a9be-cf28b38a1a7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:25:53.837874 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.837845    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-socket" pod "81ce7003-30a0-4d13-a9be-cf28b38a1a7d" (UID: "81ce7003-30a0-4d13-a9be-cf28b38a1a7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:53.837935 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.837910    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-socket\") pod \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\" (UID: \"81ce7003-30a0-4d13-a9be-cf28b38a1a7d\") "
Nov 01 04:25:53.838523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.838503    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:53.838639 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.838529    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:53.838639 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.838546    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:53.838639 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.838559    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:53.838639 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.838572    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:53.839507 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.839478    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-token" pod "81ce7003-30a0-4d13-a9be-cf28b38a1a7d" (UID: "81ce7003-30a0-4d13-a9be-cf28b38a1a7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:25:53.839736 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.839704    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-podinfo" pod "81ce7003-30a0-4d13-a9be-cf28b38a1a7d" (UID: "81ce7003-30a0-4d13-a9be-cf28b38a1a7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:25:53.839839 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.839733    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-kube-api-access-pdz8f" pod "81ce7003-30a0-4d13-a9be-cf28b38a1a7d" (UID: "81ce7003-30a0-4d13-a9be-cf28b38a1a7d"). InnerVolumeSpecName "kube-api-access-pdz8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:25:53.839839 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.839791    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-envoy" pod "81ce7003-30a0-4d13-a9be-cf28b38a1a7d" (UID: "81ce7003-30a0-4d13-a9be-cf28b38a1a7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:25:53.939562 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.939531    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:53.939562 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.939555    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:53.939562 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.939566    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdz8f\" (UniqueName: \"kubernetes.io/projected/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-kube-api-access-pdz8f\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:53.939797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:53.939576    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ce7003-30a0-4d13-a9be-cf28b38a1a7d-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:54.163747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.163667    4882 generic.go:356] "Generic (PLEG): container finished" podID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerID="3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827" exitCode=0
Nov 01 04:25:54.163747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.163731    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"
Nov 01 04:25:54.164177 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.163751    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" event={"ID":"81ce7003-30a0-4d13-a9be-cf28b38a1a7d","Type":"ContainerDied","Data":"3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827"}
Nov 01 04:25:54.164177 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.163802    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9" event={"ID":"81ce7003-30a0-4d13-a9be-cf28b38a1a7d","Type":"ContainerDied","Data":"6f6cfd3453f0a3b37c97c700c66ae6c787cb3edbdf284b2f103dcd6811271e8f"}
Nov 01 04:25:54.164177 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.163831    4882 scope.go:122] "RemoveContainer" containerID="3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827"
Nov 01 04:25:54.179703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.177623    4882 scope.go:122] "RemoveContainer" containerID="0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595"
Nov 01 04:25:54.189519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.189501    4882 scope.go:122] "RemoveContainer" containerID="3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827"
Nov 01 04:25:54.189804 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:25:54.189782    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827\": container with ID starting with 3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827 not found: ID does not exist" containerID="3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827"
Nov 01 04:25:54.189909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.189825    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827"} err="failed to get container status \"3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827\": rpc error: code = NotFound desc = could not find container \"3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827\": container with ID starting with 3037ec67f9ebf8a2f0206cb9986e3810af393575f38ab311766bb01121bec827 not found: ID does not exist"
Nov 01 04:25:54.189909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.189852    4882 scope.go:122] "RemoveContainer" containerID="0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595"
Nov 01 04:25:54.190181 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:25:54.190159    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595\": container with ID starting with 0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595 not found: ID does not exist" containerID="0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595"
Nov 01 04:25:54.190229 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.190187    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595"} err="failed to get container status \"0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595\": rpc error: code = NotFound desc = could not find container \"0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595\": container with ID starting with 0886c32a036e942e80eaf69da54bab108efc26b931be16f97484c14eba0f6595 not found: ID does not exist"
Nov 01 04:25:54.291119 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.291097    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:54.319693 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.319671    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"]
Nov 01 04:25:54.365106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.365079    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-862ad1-openshift-default-6fb69d8785-8m2s9"]
Nov 01 04:25:54.445020 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.444980    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a9a36c-fe33-4582-959f-02b39fce15fb-kube-api-access-gmk76\" (UniqueName: \"kubernetes.io/projected/43a9a36c-fe33-4582-959f-02b39fce15fb-kube-api-access-gmk76\") pod \"43a9a36c-fe33-4582-959f-02b39fce15fb\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") "
Nov 01 04:25:54.445205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.445108    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-xks-ca\") pod \"43a9a36c-fe33-4582-959f-02b39fce15fb\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") "
Nov 01 04:25:54.445285 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.445244    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-openshift-ca\") pod \"43a9a36c-fe33-4582-959f-02b39fce15fb\" (UID: \"43a9a36c-fe33-4582-959f-02b39fce15fb\") "
Nov 01 04:25:54.445644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.445614    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-openshift-ca" pod "43a9a36c-fe33-4582-959f-02b39fce15fb" (UID: "43a9a36c-fe33-4582-959f-02b39fce15fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:25:54.447695 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.447670    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-xks-ca" pod "43a9a36c-fe33-4582-959f-02b39fce15fb" (UID: "43a9a36c-fe33-4582-959f-02b39fce15fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:25:54.447823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.447678    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a9a36c-fe33-4582-959f-02b39fce15fb-kube-api-access-gmk76" pod "43a9a36c-fe33-4582-959f-02b39fce15fb" (UID: "43a9a36c-fe33-4582-959f-02b39fce15fb"). InnerVolumeSpecName "kube-api-access-gmk76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:25:54.547101 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.547066    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:54.547101 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.547093    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a9a36c-fe33-4582-959f-02b39fce15fb-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:54.547300 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:54.547107    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmk76\" (UniqueName: \"kubernetes.io/projected/43a9a36c-fe33-4582-959f-02b39fce15fb-kube-api-access-gmk76\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:25:55.149830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:55.149788    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"]
Nov 01 04:25:55.172673 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:55.172643    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"
Nov 01 04:25:55.173138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:55.172676    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x" event={"ID":"43a9a36c-fe33-4582-959f-02b39fce15fb","Type":"ContainerDied","Data":"d798604186462ddb21c3ebbacda7a8ca5636e37f6bcc23bc5ffa91d4b89cdc7f"}
Nov 01 04:25:55.173138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:55.172724    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d798604186462ddb21c3ebbacda7a8ca5636e37f6bcc23bc5ffa91d4b89cdc7f"
Nov 01 04:25:55.599826 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:25:55.599794    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" path="/var/lib/kubelet/pods/81ce7003-30a0-4d13-a9be-cf28b38a1a7d/volumes"
Nov 01 04:25:58.723341 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:25:58.723292    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:26:10.170485 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.170416    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" containerID="cri-o://e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e" gracePeriod=45
Nov 01 04:26:10.429728 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.429667    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:26:10.510834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.510805    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-tmp-dir\") pod \"210fc38c-871d-4b42-8e35-e3407510f122\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") "
Nov 01 04:26:10.511000 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.510853    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210fc38c-871d-4b42-8e35-e3407510f122-kube-api-access-rnhmf\" (UniqueName: \"kubernetes.io/projected/210fc38c-871d-4b42-8e35-e3407510f122-kube-api-access-rnhmf\") pod \"210fc38c-871d-4b42-8e35-e3407510f122\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") "
Nov 01 04:26:10.511000 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.510904    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-dshm\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-dshm\") pod \"210fc38c-871d-4b42-8e35-e3407510f122\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") "
Nov 01 04:26:10.511000 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.510958    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/210fc38c-871d-4b42-8e35-e3407510f122-tls-certs\" (UniqueName: \"kubernetes.io/secret/210fc38c-871d-4b42-8e35-e3407510f122-tls-certs\") pod \"210fc38c-871d-4b42-8e35-e3407510f122\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") "
Nov 01 04:26:10.511250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.511003    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-home\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-home\") pod \"210fc38c-871d-4b42-8e35-e3407510f122\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") "
Nov 01 04:26:10.511250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.511035    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-model-cache\") pod \"210fc38c-871d-4b42-8e35-e3407510f122\" (UID: \"210fc38c-871d-4b42-8e35-e3407510f122\") "
Nov 01 04:26:10.511250 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.511060    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-tmp-dir" pod "210fc38c-871d-4b42-8e35-e3407510f122" (UID: "210fc38c-871d-4b42-8e35-e3407510f122"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:26:10.511412 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.511296    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-home" pod "210fc38c-871d-4b42-8e35-e3407510f122" (UID: "210fc38c-871d-4b42-8e35-e3407510f122"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:26:10.511412 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.511328    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-model-cache" pod "210fc38c-871d-4b42-8e35-e3407510f122" (UID: "210fc38c-871d-4b42-8e35-e3407510f122"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:26:10.511541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.511519    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-home\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:10.511588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.511541    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-model-cache\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:10.511588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.511556    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-tmp-dir\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:10.513311 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.513280    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210fc38c-871d-4b42-8e35-e3407510f122-kube-api-access-rnhmf" pod "210fc38c-871d-4b42-8e35-e3407510f122" (UID: "210fc38c-871d-4b42-8e35-e3407510f122"). InnerVolumeSpecName "kube-api-access-rnhmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:26:10.513453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.513322    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210fc38c-871d-4b42-8e35-e3407510f122-tls-certs" pod "210fc38c-871d-4b42-8e35-e3407510f122" (UID: "210fc38c-871d-4b42-8e35-e3407510f122"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:26:10.513453 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.513327    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-dshm" pod "210fc38c-871d-4b42-8e35-e3407510f122" (UID: "210fc38c-871d-4b42-8e35-e3407510f122"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:26:10.612819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.612742    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210fc38c-871d-4b42-8e35-e3407510f122-dshm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:10.612819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.612777    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210fc38c-871d-4b42-8e35-e3407510f122-tls-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:10.612819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:10.612793    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnhmf\" (UniqueName: \"kubernetes.io/projected/210fc38c-871d-4b42-8e35-e3407510f122-kube-api-access-rnhmf\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:11.268840 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.268807    4882 generic.go:356] "Generic (PLEG): container finished" podID="210fc38c-871d-4b42-8e35-e3407510f122" containerID="e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e" exitCode=0
Nov 01 04:26:11.269277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.268852    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"
Nov 01 04:26:11.269277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.268889    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" event={"ID":"210fc38c-871d-4b42-8e35-e3407510f122","Type":"ContainerDied","Data":"e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e"}
Nov 01 04:26:11.269277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.268933    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" event={"ID":"210fc38c-871d-4b42-8e35-e3407510f122","Type":"ContainerDied","Data":"2f0956e93e52d2a5c4549af78d1104153c9726f1b2411099d7011ce0a99af1e9"}
Nov 01 04:26:11.269277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.268956    4882 scope.go:122] "RemoveContainer" containerID="e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e"
Nov 01 04:26:11.279460 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.279438    4882 scope.go:122] "RemoveContainer" containerID="e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e"
Nov 01 04:26:11.279722 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:26:11.279705    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e\": container with ID starting with e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e not found: ID does not exist" containerID="e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e"
Nov 01 04:26:11.279779 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.279729    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e"} err="failed to get container status \"e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e\": rpc error: code = NotFound desc = could not find container \"e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e\": container with ID starting with e9fbd60e199372561c377564dbdbb5d582e0b93ee4010fd479e3ce659ae56a5e not found: ID does not exist"
Nov 01 04:26:11.380811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.380776    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:26:11.453867 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.453834    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"]
Nov 01 04:26:11.482217 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.482184    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-w0-162191-kserve-6695bf8cdf-8w8tw"]
Nov 01 04:26:11.599889 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.599816    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210fc38c-871d-4b42-8e35-e3407510f122" path="/var/lib/kubelet/pods/210fc38c-871d-4b42-8e35-e3407510f122/volumes"
Nov 01 04:26:11.808799 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.808768    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"]
Nov 01 04:26:11.809239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809225    4882 cpu_manager.go:414] "containerMap: removing container" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy"
Nov 01 04:26:11.809286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809240    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy"
Nov 01 04:26:11.809286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809252    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy"
Nov 01 04:26:11.809286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809257    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f50a6fa4-73bc-42b8-a250-6f12d7e2bf9b" containerName="istio-proxy"
Nov 01 04:26:11.809286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809274    4882 cpu_manager.go:414] "containerMap: removing container" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main"
Nov 01 04:26:11.809286 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809279    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main"
Nov 01 04:26:11.809478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809295    4882 cpu_manager.go:414] "containerMap: removing container" podUID="43a9a36c-fe33-4582-959f-02b39fce15fb" containerName="revoke-keys"
Nov 01 04:26:11.809478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809301    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a9a36c-fe33-4582-959f-02b39fce15fb" containerName="revoke-keys"
Nov 01 04:26:11.809478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809386    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a9a36c-fe33-4582-959f-02b39fce15fb" containerName="revoke-keys"
Nov 01 04:26:11.809478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809397    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy"
Nov 01 04:26:11.809478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809404    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy"
Nov 01 04:26:11.809478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.809413    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="210fc38c-871d-4b42-8e35-e3407510f122" containerName="main"
Nov 01 04:26:11.814183 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.814163    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.824237 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.824211    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-envoy\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.824355 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.824242    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istiod-ca-cert\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.824355 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.824264    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4wqk\" (UniqueName: \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-kube-api-access-x4wqk\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.824355 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.824337    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-podinfo\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.824586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.824442    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-data\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.824586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.824489    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-socket\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.824586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.824512    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-certs\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.824689 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.824593    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-credential-socket\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.824689 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.824648    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-token\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.862893 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.862829    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-804014-openshift-default-dockercfg-ltbkv\""
Nov 01 04:26:11.874687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.874664    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"]
Nov 01 04:26:11.925946 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.925914    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-envoy\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.925952    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istiod-ca-cert\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4wqk\" (UniqueName: \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-kube-api-access-x4wqk\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926114    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-podinfo\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926302 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926179    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-data\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926302 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926224    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-socket\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926302 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926257    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-certs\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926299    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-credential-socket\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926396    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-token\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926619 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926599    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-socket\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926676 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926663    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-certs\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926826 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926772    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istiod-ca-cert\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926907 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926860    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-credential-socket\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.926966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.926918    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-data\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.928383 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.928352    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-envoy\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.928612 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.928594    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-podinfo\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.983184 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.983154    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-token\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:11.983310 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:11.983267    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4wqk\" (UniqueName: \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-kube-api-access-x4wqk\") pod \"e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") " pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:12.127230 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:12.127124    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:12.550138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:12.550003    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"]
Nov 01 04:26:12.552951 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:26:12.552902    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4e2fcf_c53e_4b42_ad8c_e9f195fd0dac.slice/crio-c1f3f08a4f6bc46ee13815b097293ad8f7612a9afa3d90101108a1d1e2454a8b WatchSource:0}: Error finding container c1f3f08a4f6bc46ee13815b097293ad8f7612a9afa3d90101108a1d1e2454a8b: Status 404 returned error can't find the container with id c1f3f08a4f6bc46ee13815b097293ad8f7612a9afa3d90101108a1d1e2454a8b
Nov 01 04:26:12.555332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:12.555303    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:26:12.555458 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:12.555364    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:26:12.555458 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:12.555394    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:26:13.283886 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:13.283847    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" event={"ID":"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac","Type":"ContainerStarted","Data":"f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a"}
Nov 01 04:26:13.284063 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:13.283911    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" event={"ID":"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac","Type":"ContainerStarted","Data":"c1f3f08a4f6bc46ee13815b097293ad8f7612a9afa3d90101108a1d1e2454a8b"}
Nov 01 04:26:13.441199 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:13.441148    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" podStartSLOduration=2.441134203 podStartE2EDuration="2.441134203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:26:13.440874872 +0000 UTC m=+2642.002907888" watchObservedRunningTime="2026-11-01 04:26:13.441134203 +0000 UTC m=+2642.003167243"
Nov 01 04:26:13.601603 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:13.601528    4882 scope.go:122] "RemoveContainer" containerID="58fc46b431fa03455d050ba6e6831976682041dcd02581e5ca7ce65a1dbaee43"
Nov 01 04:26:14.127698 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:14.127667    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:14.132927 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:14.132902    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:14.289783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:14.289747    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:14.290895 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:14.290867    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:26:19.975408 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:19.975366    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"]
Nov 01 04:26:19.976169 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:19.976132    4882 cpu_manager.go:414] "containerMap: removing container" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy"
Nov 01 04:26:19.976169 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:19.976162    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ce7003-30a0-4d13-a9be-cf28b38a1a7d" containerName="istio-proxy"
Nov 01 04:26:19.981369 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:19.981345    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:26:20.077992 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.077962    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-804014-dockercfg-sgktm\""
Nov 01 04:26:20.078153 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.078101    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-804014\""
Nov 01 04:26:20.082052 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.082022    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"]
Nov 01 04:26:20.134031 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.133995    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"]
Nov 01 04:26:20.134708 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.134668    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4kkv\" (UniqueName: \"kubernetes.io/projected/fa8f7f58-3d22-463c-a248-8cd08860bd07-kube-api-access-t4kkv\") pod \"payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs\" (UID: \"fa8f7f58-3d22-463c-a248-8cd08860bd07\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:26:20.134832 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.134773    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8f7f58-3d22-463c-a248-8cd08860bd07-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs\" (UID: \"fa8f7f58-3d22-463c-a248-8cd08860bd07\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:26:20.138511 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.138486    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:26:20.191080 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.191055    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"]
Nov 01 04:26:20.236164 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.236091    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h29w5\" (UniqueName: \"kubernetes.io/projected/53386651-47b5-4947-8b64-5ab4bc64081d-kube-api-access-h29w5\") pod \"payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm\" (UID: \"53386651-47b5-4947-8b64-5ab4bc64081d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:26:20.236164 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.236130    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4kkv\" (UniqueName: \"kubernetes.io/projected/fa8f7f58-3d22-463c-a248-8cd08860bd07-kube-api-access-t4kkv\") pod \"payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs\" (UID: \"fa8f7f58-3d22-463c-a248-8cd08860bd07\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:26:20.236376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.236222    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53386651-47b5-4947-8b64-5ab4bc64081d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm\" (UID: \"53386651-47b5-4947-8b64-5ab4bc64081d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:26:20.236376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.236290    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8f7f58-3d22-463c-a248-8cd08860bd07-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs\" (UID: \"fa8f7f58-3d22-463c-a248-8cd08860bd07\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:26:20.236986 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.236961    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8f7f58-3d22-463c-a248-8cd08860bd07-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs\" (UID: \"fa8f7f58-3d22-463c-a248-8cd08860bd07\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:26:20.293393 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.293334    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4kkv\" (UniqueName: \"kubernetes.io/projected/fa8f7f58-3d22-463c-a248-8cd08860bd07-kube-api-access-t4kkv\") pod \"payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs\" (UID: \"fa8f7f58-3d22-463c-a248-8cd08860bd07\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:26:20.295125 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.295107    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:26:20.337815 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.337777    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29w5\" (UniqueName: \"kubernetes.io/projected/53386651-47b5-4947-8b64-5ab4bc64081d-kube-api-access-h29w5\") pod \"payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm\" (UID: \"53386651-47b5-4947-8b64-5ab4bc64081d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:26:20.337942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.337869    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53386651-47b5-4947-8b64-5ab4bc64081d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm\" (UID: \"53386651-47b5-4947-8b64-5ab4bc64081d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:26:20.338637 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.338614    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53386651-47b5-4947-8b64-5ab4bc64081d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm\" (UID: \"53386651-47b5-4947-8b64-5ab4bc64081d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:26:20.394883 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.394852    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h29w5\" (UniqueName: \"kubernetes.io/projected/53386651-47b5-4947-8b64-5ab4bc64081d-kube-api-access-h29w5\") pod \"payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm\" (UID: \"53386651-47b5-4947-8b64-5ab4bc64081d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:26:20.452875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.452841    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:26:20.667688 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.667659    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"]
Nov 01 04:26:20.820283 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:20.820258    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"]
Nov 01 04:26:20.822301 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:26:20.822267    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53386651_47b5_4947_8b64_5ab4bc64081d.slice/crio-7507a436b0bcc459ded0b69ff9a95a7cb7d58f424f3ab5b6cfcfa9414327da8c WatchSource:0}: Error finding container 7507a436b0bcc459ded0b69ff9a95a7cb7d58f424f3ab5b6cfcfa9414327da8c: Status 404 returned error can't find the container with id 7507a436b0bcc459ded0b69ff9a95a7cb7d58f424f3ab5b6cfcfa9414327da8c
Nov 01 04:26:21.334710 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:21.334676    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs" event={"ID":"fa8f7f58-3d22-463c-a248-8cd08860bd07","Type":"ContainerStarted","Data":"c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3"}
Nov 01 04:26:21.335194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:21.334732    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs" event={"ID":"fa8f7f58-3d22-463c-a248-8cd08860bd07","Type":"ContainerStarted","Data":"3239bc5f3d5b953193a16366a8c8850b033bf23d0aa875916b5f42cbd329c516"}
Nov 01 04:26:21.335194 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:21.334917    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:26:21.336586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:21.336560    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm" event={"ID":"53386651-47b5-4947-8b64-5ab4bc64081d","Type":"ContainerStarted","Data":"8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1"}
Nov 01 04:26:21.336697 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:21.336603    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm" event={"ID":"53386651-47b5-4947-8b64-5ab4bc64081d","Type":"ContainerStarted","Data":"7507a436b0bcc459ded0b69ff9a95a7cb7d58f424f3ab5b6cfcfa9414327da8c"}
Nov 01 04:26:21.336755 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:21.336703    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:26:21.442593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:21.442549    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs" podStartSLOduration=2.44253438 podStartE2EDuration="2.44253438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:26:21.441886006 +0000 UTC m=+2650.003919024" watchObservedRunningTime="2026-11-01 04:26:21.44253438 +0000 UTC m=+2650.004567397"
Nov 01 04:26:21.550479 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:21.550418    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm" podStartSLOduration=1.5504022 podStartE2EDuration="1.5504022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:26:21.549309246 +0000 UTC m=+2650.111342262" watchObservedRunningTime="2026-11-01 04:26:21.5504022 +0000 UTC m=+2650.112435218"
Nov 01 04:26:23.113995 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.113956    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"]
Nov 01 04:26:23.118112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.118090    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:23.169921 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.169876    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:26:23.170706 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.170679    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"]
Nov 01 04:26:23.174984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.174961    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:23.212269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.212239    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:26:23.215402 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.215363    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"]
Nov 01 04:26:23.246085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.246045    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"]
Nov 01 04:26:23.289450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.289106    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tglrk\" (UniqueName: \"kubernetes.io/projected/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-kube-api-access-tglrk\") pod \"maas-api-revoke-sub-a7736ecb-zs48z\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:23.289450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.289177    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7736ecb-zs48z\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:23.289450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.289221    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkxv6\" (UniqueName: \"kubernetes.io/projected/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-kube-api-access-xkxv6\") pod \"maas-api-revoke-sub-2e113b8f-t75m8\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:23.289450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.289284    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e113b8f-t75m8\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:23.289450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.289308    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7736ecb-zs48z\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:23.289830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.289502    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e113b8f-t75m8\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:23.404167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.404068    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e113b8f-t75m8\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:23.404167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.404114    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7736ecb-zs48z\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:23.404422 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.404252    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e113b8f-t75m8\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:23.404422 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.404312    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tglrk\" (UniqueName: \"kubernetes.io/projected/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-kube-api-access-tglrk\") pod \"maas-api-revoke-sub-a7736ecb-zs48z\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:23.404422 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.404357    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7736ecb-zs48z\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:23.404422 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.404403    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkxv6\" (UniqueName: \"kubernetes.io/projected/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-kube-api-access-xkxv6\") pod \"maas-api-revoke-sub-2e113b8f-t75m8\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:23.404972 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.404945    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7736ecb-zs48z\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:23.405145 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.405085    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e113b8f-t75m8\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:23.406764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.406736    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e113b8f-t75m8\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:23.406842 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.406747    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7736ecb-zs48z\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:23.466398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.466363    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tglrk\" (UniqueName: \"kubernetes.io/projected/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-kube-api-access-tglrk\") pod \"maas-api-revoke-sub-a7736ecb-zs48z\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:23.466591 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.466363    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkxv6\" (UniqueName: \"kubernetes.io/projected/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-kube-api-access-xkxv6\") pod \"maas-api-revoke-sub-2e113b8f-t75m8\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:23.493817 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.493777    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:23.733301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.733270    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:23.887590 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:23.887543    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"]
Nov 01 04:26:23.892097 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:26:23.892051    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9b81c4_51e3_4cfe_bcee_673bc419e77d.slice/crio-f2e4606efb7b7ae9e0f698f1abc3f9a98fdc8a790aadab6641b0f4a9222f80cb WatchSource:0}: Error finding container f2e4606efb7b7ae9e0f698f1abc3f9a98fdc8a790aadab6641b0f4a9222f80cb: Status 404 returned error can't find the container with id f2e4606efb7b7ae9e0f698f1abc3f9a98fdc8a790aadab6641b0f4a9222f80cb
Nov 01 04:26:24.125452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:24.125409    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"]
Nov 01 04:26:24.127022 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:26:24.126979    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533b9cfb_1cc1_4a4f_8cdd_f48d03e0e995.slice/crio-b6b65c21389217e18c7ca8b1d754e15c322dd3a2f1edc12b7ee5567dfcdff29c WatchSource:0}: Error finding container b6b65c21389217e18c7ca8b1d754e15c322dd3a2f1edc12b7ee5567dfcdff29c: Status 404 returned error can't find the container with id b6b65c21389217e18c7ca8b1d754e15c322dd3a2f1edc12b7ee5567dfcdff29c
Nov 01 04:26:24.357483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:24.357403    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8" event={"ID":"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995","Type":"ContainerStarted","Data":"b6b65c21389217e18c7ca8b1d754e15c322dd3a2f1edc12b7ee5567dfcdff29c"}
Nov 01 04:26:24.358987 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:24.358964    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z" event={"ID":"4a9b81c4-51e3-4cfe-bcee-673bc419e77d","Type":"ContainerStarted","Data":"239513fa895f48efd30d86f25ab34fe5779f1889c61c23cd74d90586076f9a9a"}
Nov 01 04:26:24.359104 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:24.359001    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z" event={"ID":"4a9b81c4-51e3-4cfe-bcee-673bc419e77d","Type":"ContainerStarted","Data":"f2e4606efb7b7ae9e0f698f1abc3f9a98fdc8a790aadab6641b0f4a9222f80cb"}
Nov 01 04:26:25.390825 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:25.390744    4882 generic.go:356] "Generic (PLEG): container finished" podID="4a9b81c4-51e3-4cfe-bcee-673bc419e77d" containerID="239513fa895f48efd30d86f25ab34fe5779f1889c61c23cd74d90586076f9a9a" exitCode=0
Nov 01 04:26:25.391252 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:25.390828    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z" event={"ID":"4a9b81c4-51e3-4cfe-bcee-673bc419e77d","Type":"ContainerDied","Data":"239513fa895f48efd30d86f25ab34fe5779f1889c61c23cd74d90586076f9a9a"}
Nov 01 04:26:25.392488 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:25.392458    4882 generic.go:356] "Generic (PLEG): container finished" podID="533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995" containerID="76350034e4b3313c6e4678d1c65a0c63ec94c5cfefd5eadada553e34ce20f2a2" exitCode=0
Nov 01 04:26:25.392617 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:25.392494    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8" event={"ID":"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995","Type":"ContainerDied","Data":"76350034e4b3313c6e4678d1c65a0c63ec94c5cfefd5eadada553e34ce20f2a2"}
Nov 01 04:26:26.614907 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.614881    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:26.628263 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.628239    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:26.739934 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.739902    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-kube-api-access-xkxv6\" (UniqueName: \"kubernetes.io/projected/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-kube-api-access-xkxv6\") pod \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") "
Nov 01 04:26:26.740179 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.739958    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-kube-api-access-tglrk\" (UniqueName: \"kubernetes.io/projected/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-kube-api-access-tglrk\") pod \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") "
Nov 01 04:26:26.740179 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.740047    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-xks-ca\") pod \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") "
Nov 01 04:26:26.740179 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.740101    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-xks-ca\") pod \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") "
Nov 01 04:26:26.740179 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.740142    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-openshift-ca\") pod \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\" (UID: \"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995\") "
Nov 01 04:26:26.740321 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.740204    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-openshift-ca\") pod \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\" (UID: \"4a9b81c4-51e3-4cfe-bcee-673bc419e77d\") "
Nov 01 04:26:26.740771 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.740707    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-openshift-ca" pod "4a9b81c4-51e3-4cfe-bcee-673bc419e77d" (UID: "4a9b81c4-51e3-4cfe-bcee-673bc419e77d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:26:26.741017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.740741    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-openshift-ca" pod "533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995" (UID: "533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:26:26.741247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.741193    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:26.741247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.741223    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:26.742866 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.742831    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-xks-ca" pod "4a9b81c4-51e3-4cfe-bcee-673bc419e77d" (UID: "4a9b81c4-51e3-4cfe-bcee-673bc419e77d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:26:26.743118 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.743096    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-kube-api-access-tglrk" pod "4a9b81c4-51e3-4cfe-bcee-673bc419e77d" (UID: "4a9b81c4-51e3-4cfe-bcee-673bc419e77d"). InnerVolumeSpecName "kube-api-access-tglrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:26:26.743279 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.743257    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-xks-ca" pod "533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995" (UID: "533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:26:26.743366 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.743350    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-kube-api-access-xkxv6" pod "533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995" (UID: "533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995"). InnerVolumeSpecName "kube-api-access-xkxv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:26:26.841959 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.841918    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:26.841959 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.841954    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkxv6\" (UniqueName: \"kubernetes.io/projected/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995-kube-api-access-xkxv6\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:26.842174 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.841971    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tglrk\" (UniqueName: \"kubernetes.io/projected/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-kube-api-access-tglrk\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:26.842174 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:26.841984    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a9b81c4-51e3-4cfe-bcee-673bc419e77d-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:27.408901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:27.408239    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z" event={"ID":"4a9b81c4-51e3-4cfe-bcee-673bc419e77d","Type":"ContainerDied","Data":"f2e4606efb7b7ae9e0f698f1abc3f9a98fdc8a790aadab6641b0f4a9222f80cb"}
Nov 01 04:26:27.408901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:27.408295    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2e4606efb7b7ae9e0f698f1abc3f9a98fdc8a790aadab6641b0f4a9222f80cb"
Nov 01 04:26:27.408901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:27.408246    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"
Nov 01 04:26:27.409589 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:27.409567    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8" event={"ID":"533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995","Type":"ContainerDied","Data":"b6b65c21389217e18c7ca8b1d754e15c322dd3a2f1edc12b7ee5567dfcdff29c"}
Nov 01 04:26:27.409648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:27.409591    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"
Nov 01 04:26:27.409648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:27.409610    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6b65c21389217e18c7ca8b1d754e15c322dd3a2f1edc12b7ee5567dfcdff29c"
Nov 01 04:26:28.144423 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.144388    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"]
Nov 01 04:26:28.145199 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.145178    4882 cpu_manager.go:414] "containerMap: removing container" podUID="533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995" containerName="revoke-keys"
Nov 01 04:26:28.145199 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.145202    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995" containerName="revoke-keys"
Nov 01 04:26:28.145302 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.145224    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4a9b81c4-51e3-4cfe-bcee-673bc419e77d" containerName="revoke-keys"
Nov 01 04:26:28.145302 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.145230    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a9b81c4-51e3-4cfe-bcee-673bc419e77d" containerName="revoke-keys"
Nov 01 04:26:28.145368 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.145320    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a9b81c4-51e3-4cfe-bcee-673bc419e77d" containerName="revoke-keys"
Nov 01 04:26:28.145368 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.145339    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995" containerName="revoke-keys"
Nov 01 04:26:28.148630 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.148609    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.155078 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.155049    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-podinfo\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.155196 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.155124    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb2rj\" (UniqueName: \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-kube-api-access-bb2rj\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.155196 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.155158    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-socket\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.155327 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.155196    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-certs\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.155327 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.155295    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-envoy\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.155456 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.155345    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-credential-socket\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.155522 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.155457    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df6b1cd-265d-417d-b4c5-13cb3247a318-istiod-ca-cert\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.155522 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.155485    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-data\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.155628 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.155544    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-token\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.197340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.197314    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f7545f-openshift-default-dockercfg-chbwt\""
Nov 01 04:26:28.248569 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.248528    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"]
Nov 01 04:26:28.256793 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.256766    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-socket\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.256932 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.256821    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-certs\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.256932 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.256860    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-envoy\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.256932 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.256897    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-credential-socket\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.257052 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.256948    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df6b1cd-265d-417d-b4c5-13cb3247a318-istiod-ca-cert\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.257052 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.256972    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-data\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.257052 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.257030    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-token\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.257272 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.257129    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-podinfo\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.257272 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.257188    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb2rj\" (UniqueName: \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-kube-api-access-bb2rj\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.257272 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.257200    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-socket\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.257726 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.257705    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-certs\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.258063 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.257765    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-credential-socket\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.258756 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.258052    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-data\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.258756 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.258553    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df6b1cd-265d-417d-b4c5-13cb3247a318-istiod-ca-cert\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.259601 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.259581    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-envoy\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.259997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.259980    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-podinfo\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.334488 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.334454    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-token\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.335694 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.335654    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb2rj\" (UniqueName: \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-kube-api-access-bb2rj\") pod \"e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") " pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.486872 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.486630    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:28.723078 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:26:28.723027    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:26:28.843819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.843741    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"]
Nov 01 04:26:28.849381 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.849357    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:28.889124 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.889080    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fc48b4-mpdsj\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:28.889291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.889131    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fc48b4-mpdsj\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:28.889291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.889260    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdv55\" (UniqueName: \"kubernetes.io/projected/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-kube-api-access-wdv55\") pod \"maas-api-revoke-sub-06fc48b4-mpdsj\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:28.898072 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.898031    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:26:28.898912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.898895    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:26:28.908309 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.908284    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"]
Nov 01 04:26:28.911267 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.911239    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"]
Nov 01 04:26:28.915471 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.915450    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:28.957901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.957874    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"]
Nov 01 04:26:28.959882 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:26:28.959841    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df6b1cd_265d_417d_b4c5_13cb3247a318.slice/crio-6c8e1c229761e3a02b09bce02739f7817ed59eb157c2d2a84c78d91bb4a7c746 WatchSource:0}: Error finding container 6c8e1c229761e3a02b09bce02739f7817ed59eb157c2d2a84c78d91bb4a7c746: Status 404 returned error can't find the container with id 6c8e1c229761e3a02b09bce02739f7817ed59eb157c2d2a84c78d91bb4a7c746
Nov 01 04:26:28.963782 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.963740    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:26:28.963879 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.963841    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:26:28.963926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.963904    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:26:28.970259 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.970238    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"]
Nov 01 04:26:28.989869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.989844    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fc48b4-mpdsj\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:28.989999 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.989893    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268bc2e6-dnvjn\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:28.990064 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.990016    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268bc2e6-dnvjn\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:28.990148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.990129    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdv55\" (UniqueName: \"kubernetes.io/projected/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-kube-api-access-wdv55\") pod \"maas-api-revoke-sub-06fc48b4-mpdsj\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:28.990200 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.990184    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7fcm\" (UniqueName: \"kubernetes.io/projected/61039e8c-9bb3-436e-9eff-b19c866b805a-kube-api-access-z7fcm\") pod \"maas-api-revoke-sub-268bc2e6-dnvjn\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:28.990400 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.990367    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fc48b4-mpdsj\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:28.991870 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.991845    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fc48b4-mpdsj\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:28.993354 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:28.993333    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fc48b4-mpdsj\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:29.047114 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.047087    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdv55\" (UniqueName: \"kubernetes.io/projected/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-kube-api-access-wdv55\") pod \"maas-api-revoke-sub-06fc48b4-mpdsj\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:29.092211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.092171    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7fcm\" (UniqueName: \"kubernetes.io/projected/61039e8c-9bb3-436e-9eff-b19c866b805a-kube-api-access-z7fcm\") pod \"maas-api-revoke-sub-268bc2e6-dnvjn\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:29.092404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.092381    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268bc2e6-dnvjn\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:29.092508 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.092481    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268bc2e6-dnvjn\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:29.093449 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.093304    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268bc2e6-dnvjn\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:29.096084 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.096019    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268bc2e6-dnvjn\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:29.148930 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.148900    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7fcm\" (UniqueName: \"kubernetes.io/projected/61039e8c-9bb3-436e-9eff-b19c866b805a-kube-api-access-z7fcm\") pod \"maas-api-revoke-sub-268bc2e6-dnvjn\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:29.167653 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.167626    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:29.234060 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.233714    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:29.430514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.430424    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" event={"ID":"9df6b1cd-265d-417d-b4c5-13cb3247a318","Type":"ContainerStarted","Data":"626db13f594b62a08c7b1ec8b52ff1edbdf55998de0fbc59234c2991199a3079"}
Nov 01 04:26:29.430514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.430509    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" event={"ID":"9df6b1cd-265d-417d-b4c5-13cb3247a318","Type":"ContainerStarted","Data":"6c8e1c229761e3a02b09bce02739f7817ed59eb157c2d2a84c78d91bb4a7c746"}
Nov 01 04:26:29.487672 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.487637    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:29.541804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.541775    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"]
Nov 01 04:26:29.543253 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.543197    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" podStartSLOduration=1.543180877 podStartE2EDuration="1.543180877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:26:29.541369767 +0000 UTC m=+2658.103402785" watchObservedRunningTime="2026-11-01 04:26:29.543180877 +0000 UTC m=+2658.105213894"
Nov 01 04:26:29.544126 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:26:29.544080    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1ea5aa_4aa0_4b2d_a7c6_17def5c877fe.slice/crio-cb04620c51b5b16dfc79dc162d0c621385017a09b0ea3a54a12284ce64ce9ec2 WatchSource:0}: Error finding container cb04620c51b5b16dfc79dc162d0c621385017a09b0ea3a54a12284ce64ce9ec2: Status 404 returned error can't find the container with id cb04620c51b5b16dfc79dc162d0c621385017a09b0ea3a54a12284ce64ce9ec2
Nov 01 04:26:29.646730 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:29.646704    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"]
Nov 01 04:26:29.648669 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:26:29.648637    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61039e8c_9bb3_436e_9eff_b19c866b805a.slice/crio-8c12986d34e237f154d5e0d1ac7ea11c1f95646dc77abb2a9991db4234d3747f WatchSource:0}: Error finding container 8c12986d34e237f154d5e0d1ac7ea11c1f95646dc77abb2a9991db4234d3747f: Status 404 returned error can't find the container with id 8c12986d34e237f154d5e0d1ac7ea11c1f95646dc77abb2a9991db4234d3747f
Nov 01 04:26:30.447687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:30.447644    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn" event={"ID":"61039e8c-9bb3-436e-9eff-b19c866b805a","Type":"ContainerStarted","Data":"e4c6bab24d162ef5525c0243e4bdbec024215a32e35cc87df6257eb7f8e0dc58"}
Nov 01 04:26:30.448276 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:30.448247    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn" event={"ID":"61039e8c-9bb3-436e-9eff-b19c866b805a","Type":"ContainerStarted","Data":"8c12986d34e237f154d5e0d1ac7ea11c1f95646dc77abb2a9991db4234d3747f"}
Nov 01 04:26:30.450019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:30.449993    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj" event={"ID":"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe","Type":"ContainerStarted","Data":"c0ed03aca37557763492c51bcb4eac515913ec020731f5d6da51e02529e238ac"}
Nov 01 04:26:30.450138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:30.450050    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj" event={"ID":"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe","Type":"ContainerStarted","Data":"cb04620c51b5b16dfc79dc162d0c621385017a09b0ea3a54a12284ce64ce9ec2"}
Nov 01 04:26:30.493906 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:30.493866    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:30.555654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:30.555588    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn" podStartSLOduration=2.555569819 podStartE2EDuration="2.555569819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:26:30.554173029 +0000 UTC m=+2659.116206045" watchObservedRunningTime="2026-11-01 04:26:30.555569819 +0000 UTC m=+2659.117602838"
Nov 01 04:26:30.663385 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:30.663321    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj" podStartSLOduration=2.6633078660000002 podStartE2EDuration="2.663307866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:26:30.661078073 +0000 UTC m=+2659.223111090" watchObservedRunningTime="2026-11-01 04:26:30.663307866 +0000 UTC m=+2659.225340874"
Nov 01 04:26:31.457555 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:31.457519    4882 generic.go:356] "Generic (PLEG): container finished" podID="61039e8c-9bb3-436e-9eff-b19c866b805a" containerID="e4c6bab24d162ef5525c0243e4bdbec024215a32e35cc87df6257eb7f8e0dc58" exitCode=0
Nov 01 04:26:31.458030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:31.457608    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn" event={"ID":"61039e8c-9bb3-436e-9eff-b19c866b805a","Type":"ContainerDied","Data":"e4c6bab24d162ef5525c0243e4bdbec024215a32e35cc87df6257eb7f8e0dc58"}
Nov 01 04:26:31.459297 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:31.459268    4882 generic.go:356] "Generic (PLEG): container finished" podID="3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe" containerID="c0ed03aca37557763492c51bcb4eac515913ec020731f5d6da51e02529e238ac" exitCode=0
Nov 01 04:26:31.459414 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:31.459353    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj" event={"ID":"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe","Type":"ContainerDied","Data":"c0ed03aca37557763492c51bcb4eac515913ec020731f5d6da51e02529e238ac"}
Nov 01 04:26:31.459586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:31.459569    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:31.460686 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:31.460669    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:26:32.345944 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.345915    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:26:32.346142 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.345982    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:26:32.669447 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.669399    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:32.673698 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.673677    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:32.734000 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.733967    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-kube-api-access-wdv55\" (UniqueName: \"kubernetes.io/projected/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-kube-api-access-wdv55\") pod \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") "
Nov 01 04:26:32.734185 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.734048    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-xks-ca\") pod \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") "
Nov 01 04:26:32.734185 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.734119    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-xks-ca\") pod \"61039e8c-9bb3-436e-9eff-b19c866b805a\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") "
Nov 01 04:26:32.734185 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.734158    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-openshift-ca\") pod \"61039e8c-9bb3-436e-9eff-b19c866b805a\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") "
Nov 01 04:26:32.734393 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.734193    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61039e8c-9bb3-436e-9eff-b19c866b805a-kube-api-access-z7fcm\" (UniqueName: \"kubernetes.io/projected/61039e8c-9bb3-436e-9eff-b19c866b805a-kube-api-access-z7fcm\") pod \"61039e8c-9bb3-436e-9eff-b19c866b805a\" (UID: \"61039e8c-9bb3-436e-9eff-b19c866b805a\") "
Nov 01 04:26:32.734393 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.734235    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-openshift-ca\") pod \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\" (UID: \"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe\") "
Nov 01 04:26:32.734640 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.734606    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-openshift-ca" pod "61039e8c-9bb3-436e-9eff-b19c866b805a" (UID: "61039e8c-9bb3-436e-9eff-b19c866b805a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:26:32.734783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.734677    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-openshift-ca" pod "3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe" (UID: "3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:26:32.734783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.734702    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:32.736493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.736467    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-kube-api-access-wdv55" pod "3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe" (UID: "3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe"). InnerVolumeSpecName "kube-api-access-wdv55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:26:32.736622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.736521    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-xks-ca" pod "3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe" (UID: "3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:26:32.736764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.736741    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-xks-ca" pod "61039e8c-9bb3-436e-9eff-b19c866b805a" (UID: "61039e8c-9bb3-436e-9eff-b19c866b805a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:26:32.736983 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.736964    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61039e8c-9bb3-436e-9eff-b19c866b805a-kube-api-access-z7fcm" pod "61039e8c-9bb3-436e-9eff-b19c866b805a" (UID: "61039e8c-9bb3-436e-9eff-b19c866b805a"). InnerVolumeSpecName "kube-api-access-z7fcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:26:32.836113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.836078    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdv55\" (UniqueName: \"kubernetes.io/projected/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-kube-api-access-wdv55\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:32.836113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.836109    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:32.836113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.836119    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61039e8c-9bb3-436e-9eff-b19c866b805a-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:32.836361 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.836128    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7fcm\" (UniqueName: \"kubernetes.io/projected/61039e8c-9bb3-436e-9eff-b19c866b805a-kube-api-access-z7fcm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:32.836361 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:32.836136    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:26:33.473737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.473707    4882 generic.go:356] "Generic (PLEG): container finished" podID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerID="b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9" exitCode=0
Nov 01 04:26:33.473940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.473781    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" event={"ID":"9390f9be-7ca3-4bf4-9da5-65cdc7daba97","Type":"ContainerDied","Data":"b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9"}
Nov 01 04:26:33.474218 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.474200    4882 scope.go:122] "RemoveContainer" containerID="b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9"
Nov 01 04:26:33.475265 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.475241    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"
Nov 01 04:26:33.475364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.475276    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn" event={"ID":"61039e8c-9bb3-436e-9eff-b19c866b805a","Type":"ContainerDied","Data":"8c12986d34e237f154d5e0d1ac7ea11c1f95646dc77abb2a9991db4234d3747f"}
Nov 01 04:26:33.475364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.475322    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c12986d34e237f154d5e0d1ac7ea11c1f95646dc77abb2a9991db4234d3747f"
Nov 01 04:26:33.475769 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.475741    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:26:33.475883 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.475816    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:26:33.475883 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.475858    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:26:33.476961 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.476936    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"
Nov 01 04:26:33.477056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.477016    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj" event={"ID":"3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe","Type":"ContainerDied","Data":"cb04620c51b5b16dfc79dc162d0c621385017a09b0ea3a54a12284ce64ce9ec2"}
Nov 01 04:26:33.477115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:33.477058    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb04620c51b5b16dfc79dc162d0c621385017a09b0ea3a54a12284ce64ce9ec2"
Nov 01 04:26:34.490374 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:34.490331    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" event={"ID":"9390f9be-7ca3-4bf4-9da5-65cdc7daba97","Type":"ContainerStarted","Data":"eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1"}
Nov 01 04:26:35.187512 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:35.187469    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:26:35.188503 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:35.188480    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 04:26:35.188658 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:35.188522    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 04:26:36.187991 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:36.187952    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 04:26:36.188415 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:36.188011    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 04:26:37.188452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.187685    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"]
Nov 01 04:26:37.188981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.188473    4882 cpu_manager.go:414] "containerMap: removing container" podUID="61039e8c-9bb3-436e-9eff-b19c866b805a" containerName="revoke-keys"
Nov 01 04:26:37.188981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.188496    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61039e8c-9bb3-436e-9eff-b19c866b805a" containerName="revoke-keys"
Nov 01 04:26:37.188981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.188567    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe" containerName="revoke-keys"
Nov 01 04:26:37.188981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.188576    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe" containerName="revoke-keys"
Nov 01 04:26:37.188981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.188674    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe" containerName="revoke-keys"
Nov 01 04:26:37.188981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.188684    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="61039e8c-9bb3-436e-9eff-b19c866b805a" containerName="revoke-keys"
Nov 01 04:26:37.189331 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.189004    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 04:26:37.189331 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.189073    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 04:26:37.192481 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.192109    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:26:37.210550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.210525    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f6e706-64f9-4e0d-87bd-50f3061fe651-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz\" (UID: \"c5f6e706-64f9-4e0d-87bd-50f3061fe651\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:26:37.210669 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.210564    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c744c\" (UniqueName: \"kubernetes.io/projected/c5f6e706-64f9-4e0d-87bd-50f3061fe651-kube-api-access-c744c\") pod \"payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz\" (UID: \"c5f6e706-64f9-4e0d-87bd-50f3061fe651\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:26:37.286766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.286731    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f7545f\""
Nov 01 04:26:37.286996 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.286761    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f7545f-dockercfg-rnkxq\""
Nov 01 04:26:37.291821 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.291792    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"]
Nov 01 04:26:37.311723 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.311693    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f6e706-64f9-4e0d-87bd-50f3061fe651-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz\" (UID: \"c5f6e706-64f9-4e0d-87bd-50f3061fe651\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:26:37.311904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.311734    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c744c\" (UniqueName: \"kubernetes.io/projected/c5f6e706-64f9-4e0d-87bd-50f3061fe651-kube-api-access-c744c\") pod \"payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz\" (UID: \"c5f6e706-64f9-4e0d-87bd-50f3061fe651\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:26:37.312500 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.312480    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f6e706-64f9-4e0d-87bd-50f3061fe651-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz\" (UID: \"c5f6e706-64f9-4e0d-87bd-50f3061fe651\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:26:37.387232 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.387190    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c744c\" (UniqueName: \"kubernetes.io/projected/c5f6e706-64f9-4e0d-87bd-50f3061fe651-kube-api-access-c744c\") pod \"payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz\" (UID: \"c5f6e706-64f9-4e0d-87bd-50f3061fe651\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:26:37.506243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.506146    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:26:37.896011 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:37.895985    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"]
Nov 01 04:26:37.898213 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:26:37.898180    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f6e706_64f9_4e0d_87bd_50f3061fe651.slice/crio-8f27a87382ec5cad8198a76f257687b2cf8047640c253eb83b0a902ca5d21455 WatchSource:0}: Error finding container 8f27a87382ec5cad8198a76f257687b2cf8047640c253eb83b0a902ca5d21455: Status 404 returned error can't find the container with id 8f27a87382ec5cad8198a76f257687b2cf8047640c253eb83b0a902ca5d21455
Nov 01 04:26:38.187972 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:38.187934    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 04:26:38.188148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:38.188000    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 04:26:38.517293 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:38.517201    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz" event={"ID":"c5f6e706-64f9-4e0d-87bd-50f3061fe651","Type":"ContainerStarted","Data":"451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314"}
Nov 01 04:26:38.517757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:38.517302    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz" event={"ID":"c5f6e706-64f9-4e0d-87bd-50f3061fe651","Type":"ContainerStarted","Data":"8f27a87382ec5cad8198a76f257687b2cf8047640c253eb83b0a902ca5d21455"}
Nov 01 04:26:38.517757 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:38.517331    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:26:38.645076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:38.645023    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz" podStartSLOduration=1.645011368 podStartE2EDuration="1.645011368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:26:38.643046697 +0000 UTC m=+2667.205079711" watchObservedRunningTime="2026-11-01 04:26:38.645011368 +0000 UTC m=+2667.207044385"
Nov 01 04:26:38.746733 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:38.746698    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:26:39.187888 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:39.187857    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 01 04:26:39.188074 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:39.187919    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 01 04:26:40.191333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:40.191302    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:26:40.192382 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:40.192361    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:26:49.528961 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:26:49.528934    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:26:58.722387 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:26:58.722350    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:27:04.891975 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.891940    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"]
Nov 01 04:27:04.897631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.897608    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:04.925712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.925683    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-data\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:04.925869 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.925739    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-envoy\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:04.925923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.925866    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fbb559-853d-47f1-a46a-347db176506c-istiod-ca-cert\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:04.925923 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.925899    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-credential-socket\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:04.926000 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.925922    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-certs\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:04.926042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.926007    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-istio-token\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:04.926079 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.926051    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx75s\" (UniqueName: \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-kube-api-access-vx75s\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:04.926147 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.926133    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fbb559-853d-47f1-a46a-347db176506c-istio-podinfo\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:04.926196 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.926187    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-socket\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:04.946721 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.946690    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b8319b-openshift-default-dockercfg-wj5k6\""
Nov 01 04:27:04.996135 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:04.996104    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"]
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.026937    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-envoy\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.027024    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fbb559-853d-47f1-a46a-347db176506c-istiod-ca-cert\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.027057    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-credential-socket\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.027084    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-certs\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.027124    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-istio-token\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.027171    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx75s\" (UniqueName: \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-kube-api-access-vx75s\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.027270    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fbb559-853d-47f1-a46a-347db176506c-istio-podinfo\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.027361    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-socket\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.027415    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-data\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.027587    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-credential-socket\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.027875    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-data\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.027989    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-certs\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.028117    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-socket\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.030593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.028569    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fbb559-853d-47f1-a46a-347db176506c-istiod-ca-cert\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.075583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.075512    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fbb559-853d-47f1-a46a-347db176506c-istio-podinfo\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.086901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.086869    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-envoy\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.093898 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.093863    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx75s\" (UniqueName: \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-kube-api-access-vx75s\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.094871 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.094845    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-istio-token\") pod \"e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") " pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.211108 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.211079    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:05.632256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.632224    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"]
Nov 01 04:27:05.634274 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:27:05.634243    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fbb559_853d_47f1_a46a_347db176506c.slice/crio-9c1e6e03fc75b5a272c057c696d999b907d75c2ef2ade6a8d76f5e6f2d6cd89f WatchSource:0}: Error finding container 9c1e6e03fc75b5a272c057c696d999b907d75c2ef2ade6a8d76f5e6f2d6cd89f: Status 404 returned error can't find the container with id 9c1e6e03fc75b5a272c057c696d999b907d75c2ef2ade6a8d76f5e6f2d6cd89f
Nov 01 04:27:05.636780 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.636743    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:27:05.636890 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.636826    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:27:05.636954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.636886    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:27:05.723868 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.723826    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" event={"ID":"91fbb559-853d-47f1-a46a-347db176506c","Type":"ContainerStarted","Data":"81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9"}
Nov 01 04:27:05.724011 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.723891    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" event={"ID":"91fbb559-853d-47f1-a46a-347db176506c","Type":"ContainerStarted","Data":"9c1e6e03fc75b5a272c057c696d999b907d75c2ef2ade6a8d76f5e6f2d6cd89f"}
Nov 01 04:27:05.845783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:05.845733    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" podStartSLOduration=1.845716788 podStartE2EDuration="1.845716788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:27:05.844190588 +0000 UTC m=+2694.406223602" watchObservedRunningTime="2026-11-01 04:27:05.845716788 +0000 UTC m=+2694.407749805"
Nov 01 04:27:06.211498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:06.211464    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:06.217393 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:06.217370    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:06.731602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:06.731571    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:06.732652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:06.732629    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:27:12.329316 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.329281    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"]
Nov 01 04:27:12.333617 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.333594    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:27:12.425450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.425165    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e9aad9-b32e-438e-a2ec-748a6443f58e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v\" (UID: \"e5e9aad9-b32e-438e-a2ec-748a6443f58e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:27:12.425450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.425305    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzs2\" (UniqueName: \"kubernetes.io/projected/e5e9aad9-b32e-438e-a2ec-748a6443f58e-kube-api-access-mhzs2\") pod \"payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v\" (UID: \"e5e9aad9-b32e-438e-a2ec-748a6443f58e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:27:12.447752 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.447699    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b8319b-dockercfg-wlbfd\""
Nov 01 04:27:12.458936 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.458900    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"]
Nov 01 04:27:12.460561 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.460514    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b8319b\""
Nov 01 04:27:12.496550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.496519    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"]
Nov 01 04:27:12.500796 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.500779    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:27:12.526296 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.526266    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzs2\" (UniqueName: \"kubernetes.io/projected/e5e9aad9-b32e-438e-a2ec-748a6443f58e-kube-api-access-mhzs2\") pod \"payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v\" (UID: \"e5e9aad9-b32e-438e-a2ec-748a6443f58e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:27:12.526466 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.526448    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e9aad9-b32e-438e-a2ec-748a6443f58e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v\" (UID: \"e5e9aad9-b32e-438e-a2ec-748a6443f58e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:27:12.527144 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.527121    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e9aad9-b32e-438e-a2ec-748a6443f58e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v\" (UID: \"e5e9aad9-b32e-438e-a2ec-748a6443f58e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:27:12.553464 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.553423    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"]
Nov 01 04:27:12.583087 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.583016    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzs2\" (UniqueName: \"kubernetes.io/projected/e5e9aad9-b32e-438e-a2ec-748a6443f58e-kube-api-access-mhzs2\") pod \"payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v\" (UID: \"e5e9aad9-b32e-438e-a2ec-748a6443f58e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:27:12.627356 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.627325    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7kr6\" (UniqueName: \"kubernetes.io/projected/a07e6f0c-683d-44ae-9e25-db523c29b6de-kube-api-access-d7kr6\") pod \"payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x\" (UID: \"a07e6f0c-683d-44ae-9e25-db523c29b6de\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:27:12.627522 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.627368    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07e6f0c-683d-44ae-9e25-db523c29b6de-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x\" (UID: \"a07e6f0c-683d-44ae-9e25-db523c29b6de\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:27:12.647064 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.647041    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:27:12.728196 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.728162    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7kr6\" (UniqueName: \"kubernetes.io/projected/a07e6f0c-683d-44ae-9e25-db523c29b6de-kube-api-access-d7kr6\") pod \"payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x\" (UID: \"a07e6f0c-683d-44ae-9e25-db523c29b6de\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:27:12.728332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.728213    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07e6f0c-683d-44ae-9e25-db523c29b6de-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x\" (UID: \"a07e6f0c-683d-44ae-9e25-db523c29b6de\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:27:12.728919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.728899    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07e6f0c-683d-44ae-9e25-db523c29b6de-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x\" (UID: \"a07e6f0c-683d-44ae-9e25-db523c29b6de\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:27:12.784929 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.784898    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7kr6\" (UniqueName: \"kubernetes.io/projected/a07e6f0c-683d-44ae-9e25-db523c29b6de-kube-api-access-d7kr6\") pod \"payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x\" (UID: \"a07e6f0c-683d-44ae-9e25-db523c29b6de\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:27:12.814189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:12.814160    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:27:13.017908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.017879    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"]
Nov 01 04:27:13.019144 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:27:13.019107    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e9aad9_b32e_438e_a2ec_748a6443f58e.slice/crio-dc6f832ae8d829e802219483c963548f765c974eb67bb5c55a3051b2ed53ef62 WatchSource:0}: Error finding container dc6f832ae8d829e802219483c963548f765c974eb67bb5c55a3051b2ed53ef62: Status 404 returned error can't find the container with id dc6f832ae8d829e802219483c963548f765c974eb67bb5c55a3051b2ed53ef62
Nov 01 04:27:13.080640 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.080618    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:27:13.086539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.086521    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:27:13.181954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.181933    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"]
Nov 01 04:27:13.184187 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:27:13.184155    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07e6f0c_683d_44ae_9e25_db523c29b6de.slice/crio-9917f2dae8dc3671d4dfd1e7e4cf67bede57c7f493a0eb39f6698ab8bbf52935 WatchSource:0}: Error finding container 9917f2dae8dc3671d4dfd1e7e4cf67bede57c7f493a0eb39f6698ab8bbf52935: Status 404 returned error can't find the container with id 9917f2dae8dc3671d4dfd1e7e4cf67bede57c7f493a0eb39f6698ab8bbf52935
Nov 01 04:27:13.780316 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.780273    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x" event={"ID":"a07e6f0c-683d-44ae-9e25-db523c29b6de","Type":"ContainerStarted","Data":"9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13"}
Nov 01 04:27:13.780875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.780482    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:27:13.780875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.780523    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x" event={"ID":"a07e6f0c-683d-44ae-9e25-db523c29b6de","Type":"ContainerStarted","Data":"9917f2dae8dc3671d4dfd1e7e4cf67bede57c7f493a0eb39f6698ab8bbf52935"}
Nov 01 04:27:13.782203 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.782177    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v" event={"ID":"e5e9aad9-b32e-438e-a2ec-748a6443f58e","Type":"ContainerStarted","Data":"0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862"}
Nov 01 04:27:13.782328 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.782218    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v" event={"ID":"e5e9aad9-b32e-438e-a2ec-748a6443f58e","Type":"ContainerStarted","Data":"dc6f832ae8d829e802219483c963548f765c974eb67bb5c55a3051b2ed53ef62"}
Nov 01 04:27:13.782328 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.782269    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:27:13.887794 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.887748    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x" podStartSLOduration=1.8877350819999998 podStartE2EDuration="1.887735082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:27:13.887251233 +0000 UTC m=+2702.449284232" watchObservedRunningTime="2026-11-01 04:27:13.887735082 +0000 UTC m=+2702.449768098"
Nov 01 04:27:13.995131 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:13.995083    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v" podStartSLOduration=1.995065051 podStartE2EDuration="1.995065051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:27:13.994423755 +0000 UTC m=+2702.556456771" watchObservedRunningTime="2026-11-01 04:27:13.995065051 +0000 UTC m=+2702.557098068"
Nov 01 04:27:18.277451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:18.271684    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"]
Nov 01 04:27:18.314894 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:18.314854    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-162191-xpz8x"]
Nov 01 04:27:19.599951 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:19.599915    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a9a36c-fe33-4582-959f-02b39fce15fb" path="/var/lib/kubelet/pods/43a9a36c-fe33-4582-959f-02b39fce15fb/volumes"
Nov 01 04:27:24.793764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:24.793731    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:27:24.794320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:24.793810    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:27:24.865451 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:24.857110    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"]
Nov 01 04:27:24.887687 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:24.887641    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-f73b9f-4nq8c"]
Nov 01 04:27:25.602176 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:25.602136    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a4059da-8449-498f-a379-a5e407d52027" path="/var/lib/kubelet/pods/8a4059da-8449-498f-a379-a5e407d52027/volumes"
Nov 01 04:27:27.899882 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:27.899849    4882 generic.go:356] "Generic (PLEG): container finished" podID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerID="f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a" exitCode=0
Nov 01 04:27:27.900391 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:27.899920    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" event={"ID":"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac","Type":"ContainerDied","Data":"f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a"}
Nov 01 04:27:27.900391 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:27.900296    4882 scope.go:122] "RemoveContainer" containerID="f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a"
Nov 01 04:27:27.901779 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:27.901753    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:27:27.901849 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:27.901824    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:27:27.901889 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:27.901861    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:27:28.723964 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:27:28.723924    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:27:28.908727 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:28.908692    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" event={"ID":"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac","Type":"ContainerStarted","Data":"f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe"}
Nov 01 04:27:29.128300 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:29.128222    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:27:29.129274 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:29.129245    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 01 04:27:29.129408 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:29.129292    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 01 04:27:29.290315 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:29.290276    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:27:30.128485 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:30.128424    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 01 04:27:30.128981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:30.128506    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 01 04:27:31.128778 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:31.128736    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 01 04:27:31.129417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:31.128803    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 01 04:27:32.127716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:32.127681    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 01 04:27:32.127904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:32.127735    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 01 04:27:33.127996 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:33.127959    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 01 04:27:33.128401 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:33.128030    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 01 04:27:34.132339 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:34.132300    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:27:34.133395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:34.133373    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:27:50.060658 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:50.060571    4882 generic.go:356] "Generic (PLEG): container finished" podID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerID="626db13f594b62a08c7b1ec8b52ff1edbdf55998de0fbc59234c2991199a3079" exitCode=0
Nov 01 04:27:50.060658 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:50.060642    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" event={"ID":"9df6b1cd-265d-417d-b4c5-13cb3247a318","Type":"ContainerDied","Data":"626db13f594b62a08c7b1ec8b52ff1edbdf55998de0fbc59234c2991199a3079"}
Nov 01 04:27:50.061214 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:50.061170    4882 scope.go:122] "RemoveContainer" containerID="626db13f594b62a08c7b1ec8b52ff1edbdf55998de0fbc59234c2991199a3079"
Nov 01 04:27:50.062690 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:50.062663    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:27:50.062751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:50.062732    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:27:50.062788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:50.062766    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:27:51.090985 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:51.090950    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" event={"ID":"9df6b1cd-265d-417d-b4c5-13cb3247a318","Type":"ContainerStarted","Data":"c6fe7821f166e935eeaae51f7c354909cb86e989306b55e04decde8b18253228"}
Nov 01 04:27:51.487725 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:51.487680    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:27:51.488045 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:51.488020    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 04:27:51.488147 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:51.488092    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 04:27:52.487947 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:52.487904    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 04:27:52.488347 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:52.487967    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 04:27:53.487940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:53.487906    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 04:27:53.488128 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:53.487957    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 04:27:54.487926 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:54.487889    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 04:27:54.488129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:54.487951    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 04:27:55.487961 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:55.487924    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 04:27:55.488370 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:55.487995    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 04:27:56.490865 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:56.490831    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:27:56.491254 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:56.491147    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:27:56.491933 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:27:56.491915    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:27:58.723555 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:27:58.723520    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:28:13.659325 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:13.659293    4882 scope.go:122] "RemoveContainer" containerID="42ea6f18004fe3ac8c4c3e0c0b08f2d8ec2a444d77a7d4b44063d71380bb3064"
Nov 01 04:28:13.669119 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:13.669098    4882 scope.go:122] "RemoveContainer" containerID="5cbe3867ccbc212121c092952db6f12bb49fd52e75c355844d8ba8e71fc3e305"
Nov 01 04:28:16.295452 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.295405    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"]
Nov 01 04:28:16.301069 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.301044    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:16.349271 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.349237    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:28:16.395082 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.395043    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:28:16.398167 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.398138    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"]
Nov 01 04:28:16.468759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.468725    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:16.468759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.468758    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:16.469032 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.468782    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbbwg\" (UniqueName: \"kubernetes.io/projected/1cde266c-d725-431a-a075-0665e64f6b70-kube-api-access-fbbwg\") pod \"maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:16.570251 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.570159    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:16.570251 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.570201    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:16.570251 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.570225    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbbwg\" (UniqueName: \"kubernetes.io/projected/1cde266c-d725-431a-a075-0665e64f6b70-kube-api-access-fbbwg\") pod \"maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:16.570944 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.570916    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:16.572701 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.572681    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:16.627852 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.627816    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbbwg\" (UniqueName: \"kubernetes.io/projected/1cde266c-d725-431a-a075-0665e64f6b70-kube-api-access-fbbwg\") pod \"maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:16.914002 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:16.913913    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:17.283577 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:17.283554    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"]
Nov 01 04:28:17.286387 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:28:17.286358    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cde266c_d725_431a_a075_0665e64f6b70.slice/crio-301772a8a23d49aadd12f202c78af3acd6c3e4d8cb7b9d95b515ed1b259312d0 WatchSource:0}: Error finding container 301772a8a23d49aadd12f202c78af3acd6c3e4d8cb7b9d95b515ed1b259312d0: Status 404 returned error can't find the container with id 301772a8a23d49aadd12f202c78af3acd6c3e4d8cb7b9d95b515ed1b259312d0
Nov 01 04:28:18.287158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:18.287122    4882 generic.go:356] "Generic (PLEG): container finished" podID="1cde266c-d725-431a-a075-0665e64f6b70" containerID="091d5a18387b6a10a0c168f845b230d04f4cd0c0f143caf08bb84e2bf61801b9" exitCode=0
Nov 01 04:28:18.287630 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:18.287200    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4" event={"ID":"1cde266c-d725-431a-a075-0665e64f6b70","Type":"ContainerDied","Data":"091d5a18387b6a10a0c168f845b230d04f4cd0c0f143caf08bb84e2bf61801b9"}
Nov 01 04:28:18.287630 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:18.287248    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4" event={"ID":"1cde266c-d725-431a-a075-0665e64f6b70","Type":"ContainerStarted","Data":"301772a8a23d49aadd12f202c78af3acd6c3e4d8cb7b9d95b515ed1b259312d0"}
Nov 01 04:28:19.441576 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:19.441553    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:19.606395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:19.606321    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-xks-ca\") pod \"1cde266c-d725-431a-a075-0665e64f6b70\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") "
Nov 01 04:28:19.606395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:19.606362    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cde266c-d725-431a-a075-0665e64f6b70-kube-api-access-fbbwg\" (UniqueName: \"kubernetes.io/projected/1cde266c-d725-431a-a075-0665e64f6b70-kube-api-access-fbbwg\") pod \"1cde266c-d725-431a-a075-0665e64f6b70\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") "
Nov 01 04:28:19.606661 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:19.606525    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-openshift-ca\") pod \"1cde266c-d725-431a-a075-0665e64f6b70\" (UID: \"1cde266c-d725-431a-a075-0665e64f6b70\") "
Nov 01 04:28:19.606863 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:19.606838    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-openshift-ca" pod "1cde266c-d725-431a-a075-0665e64f6b70" (UID: "1cde266c-d725-431a-a075-0665e64f6b70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:28:19.608207 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:19.608183    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-xks-ca" pod "1cde266c-d725-431a-a075-0665e64f6b70" (UID: "1cde266c-d725-431a-a075-0665e64f6b70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:28:19.608386 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:19.608367    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cde266c-d725-431a-a075-0665e64f6b70-kube-api-access-fbbwg" pod "1cde266c-d725-431a-a075-0665e64f6b70" (UID: "1cde266c-d725-431a-a075-0665e64f6b70"). InnerVolumeSpecName "kube-api-access-fbbwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:28:19.707860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:19.707832    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:28:19.707860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:19.707857    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cde266c-d725-431a-a075-0665e64f6b70-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:28:19.708047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:19.707867    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbbwg\" (UniqueName: \"kubernetes.io/projected/1cde266c-d725-431a-a075-0665e64f6b70-kube-api-access-fbbwg\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:28:20.302299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:20.302271    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"
Nov 01 04:28:20.302519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:20.302300    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4" event={"ID":"1cde266c-d725-431a-a075-0665e64f6b70","Type":"ContainerDied","Data":"301772a8a23d49aadd12f202c78af3acd6c3e4d8cb7b9d95b515ed1b259312d0"}
Nov 01 04:28:20.302519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:20.302341    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="301772a8a23d49aadd12f202c78af3acd6c3e4d8cb7b9d95b515ed1b259312d0"
Nov 01 04:28:28.723412 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:28:28.723376    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:28:35.425467 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:35.425407    4882 generic.go:356] "Generic (PLEG): container finished" podID="91fbb559-853d-47f1-a46a-347db176506c" containerID="81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9" exitCode=0
Nov 01 04:28:35.425941 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:35.425484    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" event={"ID":"91fbb559-853d-47f1-a46a-347db176506c","Type":"ContainerDied","Data":"81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9"}
Nov 01 04:28:35.426042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:35.425938    4882 scope.go:122] "RemoveContainer" containerID="81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9"
Nov 01 04:28:35.427489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:35.427458    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:28:35.427588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:35.427527    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:28:35.427588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:35.427561    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:28:36.442183 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:36.442144    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" event={"ID":"91fbb559-853d-47f1-a46a-347db176506c","Type":"ContainerStarted","Data":"fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e"}
Nov 01 04:28:36.732072 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:36.731983    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:28:37.211475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:37.211406    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:28:37.212592 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:37.212563    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 01 04:28:37.212723 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:37.212614    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 01 04:28:38.212325 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:38.212291    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 01 04:28:38.212754 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:38.212353    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 01 04:28:39.212113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:39.212076    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 01 04:28:39.212302 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:39.212146    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 01 04:28:40.212178 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:40.212138    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 01 04:28:40.212656 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:40.212200    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 01 04:28:41.212188 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:41.212152    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 01 04:28:41.212607 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:41.212207    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 01 04:28:42.216364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:42.216331    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:28:42.217239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:42.217217    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:28:52.425538 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:52.425497    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"]
Nov 01 04:28:52.428588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:52.428559    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-550b8a-rb7d4"]
Nov 01 04:28:53.600566 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:53.600535    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cde266c-d725-431a-a075-0665e64f6b70" path="/var/lib/kubelet/pods/1cde266c-d725-431a-a075-0665e64f6b70/volumes"
Nov 01 04:28:58.724224 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:28:58.724185    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:28:59.520156 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:59.520125    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"]
Nov 01 04:28:59.520357 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:28:59.520315    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy" containerID="cri-o://97f458c9d4d40862cde63fd868ef69dfd4749c392aac59bdeb509ea8741439f5" gracePeriod=30
Nov 01 04:29:00.421675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.421640    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"]
Nov 01 04:29:00.422217 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.422202    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1cde266c-d725-431a-a075-0665e64f6b70" containerName="revoke-keys"
Nov 01 04:29:00.422264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.422219    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cde266c-d725-431a-a075-0665e64f6b70" containerName="revoke-keys"
Nov 01 04:29:00.422337 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.422328    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cde266c-d725-431a-a075-0665e64f6b70" containerName="revoke-keys"
Nov 01 04:29:00.426834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.426814    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:00.475094 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.475063    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:29:00.520801 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.520771    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:29:00.525163 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.525136    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"]
Nov 01 04:29:00.547227 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.547196    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:00.547376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.547251    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brj2j\" (UniqueName: \"kubernetes.io/projected/ed229bec-cfac-47f9-ac01-5a1c10279434-kube-api-access-brj2j\") pod \"maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:00.547376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.547351    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:00.648417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.648380    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:00.648632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.648487    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brj2j\" (UniqueName: \"kubernetes.io/projected/ed229bec-cfac-47f9-ac01-5a1c10279434-kube-api-access-brj2j\") pod \"maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:00.648632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.648525    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:00.649283 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.649251    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:00.650961 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.650932    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:00.705295 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.705258    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brj2j\" (UniqueName: \"kubernetes.io/projected/ed229bec-cfac-47f9-ac01-5a1c10279434-kube-api-access-brj2j\") pod \"maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:00.740333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:00.740310    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:01.116076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:01.116045    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"]
Nov 01 04:29:01.117892 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:29:01.117848    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded229bec_cfac_47f9_ac01_5a1c10279434.slice/crio-3b30602be5ac6eb6dabd5cf8ae4d93a2652e758c2d1ec396e393a1756689904b WatchSource:0}: Error finding container 3b30602be5ac6eb6dabd5cf8ae4d93a2652e758c2d1ec396e393a1756689904b: Status 404 returned error can't find the container with id 3b30602be5ac6eb6dabd5cf8ae4d93a2652e758c2d1ec396e393a1756689904b
Nov 01 04:29:01.659109 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:01.659016    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8" event={"ID":"ed229bec-cfac-47f9-ac01-5a1c10279434","Type":"ContainerStarted","Data":"9537da3989bdbd22714577f8783adc878cc29c2a9a1d9ce6f91ca6704772891e"}
Nov 01 04:29:01.659109 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:01.659083    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8" event={"ID":"ed229bec-cfac-47f9-ac01-5a1c10279434","Type":"ContainerStarted","Data":"3b30602be5ac6eb6dabd5cf8ae4d93a2652e758c2d1ec396e393a1756689904b"}
Nov 01 04:29:01.769868 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:01.769810    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8" podStartSLOduration=1.769796057 podStartE2EDuration="1.769796057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:29:01.766787122 +0000 UTC m=+2810.328820138" watchObservedRunningTime="2026-11-01 04:29:01.769796057 +0000 UTC m=+2810.331829073"
Nov 01 04:29:02.667442 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:02.667387    4882 generic.go:356] "Generic (PLEG): container finished" podID="ed229bec-cfac-47f9-ac01-5a1c10279434" containerID="9537da3989bdbd22714577f8783adc878cc29c2a9a1d9ce6f91ca6704772891e" exitCode=0
Nov 01 04:29:02.667922 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:02.667473    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8" event={"ID":"ed229bec-cfac-47f9-ac01-5a1c10279434","Type":"ContainerDied","Data":"9537da3989bdbd22714577f8783adc878cc29c2a9a1d9ce6f91ca6704772891e"}
Nov 01 04:29:03.827231 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:03.827202    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:03.989604 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:03.989570    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-xks-ca\") pod \"ed229bec-cfac-47f9-ac01-5a1c10279434\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") "
Nov 01 04:29:03.989604 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:03.989607    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-openshift-ca\") pod \"ed229bec-cfac-47f9-ac01-5a1c10279434\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") "
Nov 01 04:29:03.989875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:03.989632    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed229bec-cfac-47f9-ac01-5a1c10279434-kube-api-access-brj2j\" (UniqueName: \"kubernetes.io/projected/ed229bec-cfac-47f9-ac01-5a1c10279434-kube-api-access-brj2j\") pod \"ed229bec-cfac-47f9-ac01-5a1c10279434\" (UID: \"ed229bec-cfac-47f9-ac01-5a1c10279434\") "
Nov 01 04:29:03.989979 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:03.989956    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-openshift-ca" pod "ed229bec-cfac-47f9-ac01-5a1c10279434" (UID: "ed229bec-cfac-47f9-ac01-5a1c10279434"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:29:03.990068 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:03.990051    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:03.991691 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:03.991669    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-xks-ca" pod "ed229bec-cfac-47f9-ac01-5a1c10279434" (UID: "ed229bec-cfac-47f9-ac01-5a1c10279434"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:29:03.991819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:03.991795    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed229bec-cfac-47f9-ac01-5a1c10279434-kube-api-access-brj2j" pod "ed229bec-cfac-47f9-ac01-5a1c10279434" (UID: "ed229bec-cfac-47f9-ac01-5a1c10279434"). InnerVolumeSpecName "kube-api-access-brj2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:29:04.091207 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.091167    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed229bec-cfac-47f9-ac01-5a1c10279434-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:04.091207 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.091200    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brj2j\" (UniqueName: \"kubernetes.io/projected/ed229bec-cfac-47f9-ac01-5a1c10279434-kube-api-access-brj2j\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:04.683722 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.683686    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"
Nov 01 04:29:04.683937 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.683692    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8" event={"ID":"ed229bec-cfac-47f9-ac01-5a1c10279434","Type":"ContainerDied","Data":"3b30602be5ac6eb6dabd5cf8ae4d93a2652e758c2d1ec396e393a1756689904b"}
Nov 01 04:29:04.683937 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.683818    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b30602be5ac6eb6dabd5cf8ae4d93a2652e758c2d1ec396e393a1756689904b"
Nov 01 04:29:04.686084 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.686053    4882 generic.go:356] "Generic (PLEG): container finished" podID="71b18693-924b-492c-9db8-29a52ef1bb12" containerID="97f458c9d4d40862cde63fd868ef69dfd4749c392aac59bdeb509ea8741439f5" exitCode=0
Nov 01 04:29:04.686248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.686129    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" event={"ID":"71b18693-924b-492c-9db8-29a52ef1bb12","Type":"ContainerDied","Data":"97f458c9d4d40862cde63fd868ef69dfd4749c392aac59bdeb509ea8741439f5"}
Nov 01 04:29:04.686248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.686182    4882 scope.go:122] "RemoveContainer" containerID="721cb0d78fb595f35fdcbda10a830219c40705e52ec9d66ab3afb43afe565a73"
Nov 01 04:29:04.777705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.777676    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:29:04.906204 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906125    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-data\") pod \"71b18693-924b-492c-9db8-29a52ef1bb12\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") "
Nov 01 04:29:04.906204 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906168    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-istio-token\" (UniqueName: \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-istio-token\") pod \"71b18693-924b-492c-9db8-29a52ef1bb12\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") "
Nov 01 04:29:04.906204 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906188    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-credential-socket\") pod \"71b18693-924b-492c-9db8-29a52ef1bb12\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") "
Nov 01 04:29:04.906819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906252    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71b18693-924b-492c-9db8-29a52ef1bb12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b18693-924b-492c-9db8-29a52ef1bb12-istio-podinfo\") pod \"71b18693-924b-492c-9db8-29a52ef1bb12\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") "
Nov 01 04:29:04.906819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906391    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-kube-api-access-rtktw\" (UniqueName: \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-kube-api-access-rtktw\") pod \"71b18693-924b-492c-9db8-29a52ef1bb12\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") "
Nov 01 04:29:04.906819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906420    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-envoy\") pod \"71b18693-924b-492c-9db8-29a52ef1bb12\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") "
Nov 01 04:29:04.906819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906417    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-data" pod "71b18693-924b-492c-9db8-29a52ef1bb12" (UID: "71b18693-924b-492c-9db8-29a52ef1bb12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:29:04.906819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906463    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-certs\") pod \"71b18693-924b-492c-9db8-29a52ef1bb12\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") "
Nov 01 04:29:04.906819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906522    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b18693-924b-492c-9db8-29a52ef1bb12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b18693-924b-492c-9db8-29a52ef1bb12-istiod-ca-cert\") pod \"71b18693-924b-492c-9db8-29a52ef1bb12\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") "
Nov 01 04:29:04.906819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906566    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-credential-socket" pod "71b18693-924b-492c-9db8-29a52ef1bb12" (UID: "71b18693-924b-492c-9db8-29a52ef1bb12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:29:04.906819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906572    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-socket\") pod \"71b18693-924b-492c-9db8-29a52ef1bb12\" (UID: \"71b18693-924b-492c-9db8-29a52ef1bb12\") "
Nov 01 04:29:04.906819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906770    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-socket" pod "71b18693-924b-492c-9db8-29a52ef1bb12" (UID: "71b18693-924b-492c-9db8-29a52ef1bb12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:29:04.906819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.906798    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-certs" pod "71b18693-924b-492c-9db8-29a52ef1bb12" (UID: "71b18693-924b-492c-9db8-29a52ef1bb12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:29:04.907388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.907071    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b18693-924b-492c-9db8-29a52ef1bb12-istiod-ca-cert" pod "71b18693-924b-492c-9db8-29a52ef1bb12" (UID: "71b18693-924b-492c-9db8-29a52ef1bb12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:29:04.907388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.907223    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:04.907388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.907243    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b18693-924b-492c-9db8-29a52ef1bb12-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:04.907388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.907257    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:04.907388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.907272    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:04.907388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.907285    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:04.908830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.908803    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71b18693-924b-492c-9db8-29a52ef1bb12-istio-podinfo" pod "71b18693-924b-492c-9db8-29a52ef1bb12" (UID: "71b18693-924b-492c-9db8-29a52ef1bb12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:29:04.909021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.909002    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-istio-token" pod "71b18693-924b-492c-9db8-29a52ef1bb12" (UID: "71b18693-924b-492c-9db8-29a52ef1bb12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:29:04.909095 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.909014    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-kube-api-access-rtktw" pod "71b18693-924b-492c-9db8-29a52ef1bb12" (UID: "71b18693-924b-492c-9db8-29a52ef1bb12"). InnerVolumeSpecName "kube-api-access-rtktw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:29:04.909253 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:04.909236    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-envoy" pod "71b18693-924b-492c-9db8-29a52ef1bb12" (UID: "71b18693-924b-492c-9db8-29a52ef1bb12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:29:05.008389 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:05.008355    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b18693-924b-492c-9db8-29a52ef1bb12-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:05.008389 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:05.008385    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtktw\" (UniqueName: \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-kube-api-access-rtktw\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:05.008389 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:05.008398    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b18693-924b-492c-9db8-29a52ef1bb12-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:05.008661 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:05.008407    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b18693-924b-492c-9db8-29a52ef1bb12-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:05.693892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:05.693858    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"
Nov 01 04:29:05.694023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:05.693847    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q" event={"ID":"71b18693-924b-492c-9db8-29a52ef1bb12","Type":"ContainerDied","Data":"622a4e1f62c0308f7410526ad4c7935079d21304120ce6a1c6e372243b36fbe4"}
Nov 01 04:29:05.694023 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:05.694003    4882 scope.go:122] "RemoveContainer" containerID="97f458c9d4d40862cde63fd868ef69dfd4749c392aac59bdeb509ea8741439f5"
Nov 01 04:29:05.893206 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:05.893128    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"]
Nov 01 04:29:05.896800 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:05.896772    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-550b8a-openshift-default-8c677bf7-mhq7q"]
Nov 01 04:29:07.600564 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:07.600531    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" path="/var/lib/kubelet/pods/71b18693-924b-492c-9db8-29a52ef1bb12/volumes"
Nov 01 04:29:11.846000 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:11.845969    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"]
Nov 01 04:29:11.944240 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:11.944205    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"]
Nov 01 04:29:13.697141 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:13.697109    4882 scope.go:122] "RemoveContainer" containerID="daccfe94fac8699d0263f3494eeeb5bc726a6f8b5a5a2d54bec4b44321ff53ab"
Nov 01 04:29:13.707130 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:13.707109    4882 scope.go:122] "RemoveContainer" containerID="10e8026e43ecf9afe8f01a1ea7d1daf0d5cc59913915870514edf03ddd57bab9"
Nov 01 04:29:13.715510 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:13.715489    4882 scope.go:122] "RemoveContainer" containerID="ba24c9bf7bd0e7a052b1416c36b206cd46235959c5dff795ddacb971b4358a9c"
Nov 01 04:29:13.723672 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:13.723650    4882 scope.go:122] "RemoveContainer" containerID="2437d5f4b39180ce672644007cc5b1f6d788c07d1284ffc1ddf564020157b5eb"
Nov 01 04:29:16.846530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:16.846490    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm" podUID="53386651-47b5-4947-8b64-5ab4bc64081d" containerName="payload-processing" containerID="cri-o://8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1" gracePeriod=25
Nov 01 04:29:16.945238 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:16.945186    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs" podUID="fa8f7f58-3d22-463c-a248-8cd08860bd07" containerName="payload-pre-processing" containerID="cri-o://c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3" gracePeriod=25
Nov 01 04:29:17.113135 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.113063    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:29:17.161213 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.161179    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53386651-47b5-4947-8b64-5ab4bc64081d-kube-api-access-h29w5\" (UniqueName: \"kubernetes.io/projected/53386651-47b5-4947-8b64-5ab4bc64081d-kube-api-access-h29w5\") pod \"53386651-47b5-4947-8b64-5ab4bc64081d\" (UID: \"53386651-47b5-4947-8b64-5ab4bc64081d\") "
Nov 01 04:29:17.161410 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.161390    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53386651-47b5-4947-8b64-5ab4bc64081d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53386651-47b5-4947-8b64-5ab4bc64081d-plugins-config-volume\") pod \"53386651-47b5-4947-8b64-5ab4bc64081d\" (UID: \"53386651-47b5-4947-8b64-5ab4bc64081d\") "
Nov 01 04:29:17.163099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.163070    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53386651-47b5-4947-8b64-5ab4bc64081d-plugins-config-volume" pod "53386651-47b5-4947-8b64-5ab4bc64081d" (UID: "53386651-47b5-4947-8b64-5ab4bc64081d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:29:17.163940 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.163913    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53386651-47b5-4947-8b64-5ab4bc64081d-kube-api-access-h29w5" pod "53386651-47b5-4947-8b64-5ab4bc64081d" (UID: "53386651-47b5-4947-8b64-5ab4bc64081d"). InnerVolumeSpecName "kube-api-access-h29w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:29:17.213605 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.213582    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:29:17.262375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.262336    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8f7f58-3d22-463c-a248-8cd08860bd07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8f7f58-3d22-463c-a248-8cd08860bd07-plugins-config-volume\") pod \"fa8f7f58-3d22-463c-a248-8cd08860bd07\" (UID: \"fa8f7f58-3d22-463c-a248-8cd08860bd07\") "
Nov 01 04:29:17.262542 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.262486    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8f7f58-3d22-463c-a248-8cd08860bd07-kube-api-access-t4kkv\" (UniqueName: \"kubernetes.io/projected/fa8f7f58-3d22-463c-a248-8cd08860bd07-kube-api-access-t4kkv\") pod \"fa8f7f58-3d22-463c-a248-8cd08860bd07\" (UID: \"fa8f7f58-3d22-463c-a248-8cd08860bd07\") "
Nov 01 04:29:17.262785 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.262760    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8f7f58-3d22-463c-a248-8cd08860bd07-plugins-config-volume" pod "fa8f7f58-3d22-463c-a248-8cd08860bd07" (UID: "fa8f7f58-3d22-463c-a248-8cd08860bd07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:29:17.262967 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.262948    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53386651-47b5-4947-8b64-5ab4bc64081d-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:17.263038 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.262969    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa8f7f58-3d22-463c-a248-8cd08860bd07-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:17.263038 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.262980    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h29w5\" (UniqueName: \"kubernetes.io/projected/53386651-47b5-4947-8b64-5ab4bc64081d-kube-api-access-h29w5\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:17.264543 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.264518    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8f7f58-3d22-463c-a248-8cd08860bd07-kube-api-access-t4kkv" pod "fa8f7f58-3d22-463c-a248-8cd08860bd07" (UID: "fa8f7f58-3d22-463c-a248-8cd08860bd07"). InnerVolumeSpecName "kube-api-access-t4kkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:29:17.364225 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.364147    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4kkv\" (UniqueName: \"kubernetes.io/projected/fa8f7f58-3d22-463c-a248-8cd08860bd07-kube-api-access-t4kkv\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:17.784569 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.784534    4882 generic.go:356] "Generic (PLEG): container finished" podID="53386651-47b5-4947-8b64-5ab4bc64081d" containerID="8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1" exitCode=0
Nov 01 04:29:17.784766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.784583    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"
Nov 01 04:29:17.784766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.784616    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm" event={"ID":"53386651-47b5-4947-8b64-5ab4bc64081d","Type":"ContainerDied","Data":"8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1"}
Nov 01 04:29:17.784766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.784667    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm" event={"ID":"53386651-47b5-4947-8b64-5ab4bc64081d","Type":"ContainerDied","Data":"7507a436b0bcc459ded0b69ff9a95a7cb7d58f424f3ab5b6cfcfa9414327da8c"}
Nov 01 04:29:17.784766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.784693    4882 scope.go:122] "RemoveContainer" containerID="8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1"
Nov 01 04:29:17.786070 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.786047    4882 generic.go:356] "Generic (PLEG): container finished" podID="fa8f7f58-3d22-463c-a248-8cd08860bd07" containerID="c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3" exitCode=0
Nov 01 04:29:17.786189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.786091    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"
Nov 01 04:29:17.786189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.786133    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs" event={"ID":"fa8f7f58-3d22-463c-a248-8cd08860bd07","Type":"ContainerDied","Data":"c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3"}
Nov 01 04:29:17.786189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.786180    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs" event={"ID":"fa8f7f58-3d22-463c-a248-8cd08860bd07","Type":"ContainerDied","Data":"3239bc5f3d5b953193a16366a8c8850b033bf23d0aa875916b5f42cbd329c516"}
Nov 01 04:29:17.794711 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.794692    4882 scope.go:122] "RemoveContainer" containerID="8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1"
Nov 01 04:29:17.794996 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:29:17.794968    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1\": container with ID starting with 8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1 not found: ID does not exist" containerID="8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1"
Nov 01 04:29:17.795086 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.795006    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1"} err="failed to get container status \"8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1\": rpc error: code = NotFound desc = could not find container \"8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1\": container with ID starting with 8a92972d695ca2b2077a003d97984d2f5c1d6d885e7b0a8316f6d0b283983ba1 not found: ID does not exist"
Nov 01 04:29:17.795086 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.795030    4882 scope.go:122] "RemoveContainer" containerID="c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3"
Nov 01 04:29:17.803534 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.803517    4882 scope.go:122] "RemoveContainer" containerID="c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3"
Nov 01 04:29:17.803791 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:29:17.803772    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3\": container with ID starting with c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3 not found: ID does not exist" containerID="c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3"
Nov 01 04:29:17.803840 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.803798    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3"} err="failed to get container status \"c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3\": rpc error: code = NotFound desc = could not find container \"c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3\": container with ID starting with c0447ac5ecdae795b60296fd5a5da475fe31a4c4caca27ceb8b30e0c246018b3 not found: ID does not exist"
Nov 01 04:29:17.943619 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.943586    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"]
Nov 01 04:29:17.990365 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:17.990333    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-804014-6d666759c5-4sgpm"]
Nov 01 04:29:18.054503 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:18.054420    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"]
Nov 01 04:29:18.058805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:18.058776    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-804014-f8fb8f799-7slhs"]
Nov 01 04:29:19.128108 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128077    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"]
Nov 01 04:29:19.128620 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128605    4882 cpu_manager.go:414] "containerMap: removing container" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy"
Nov 01 04:29:19.128675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128621    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy"
Nov 01 04:29:19.128675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128640    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ed229bec-cfac-47f9-ac01-5a1c10279434" containerName="revoke-keys"
Nov 01 04:29:19.128675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128645    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed229bec-cfac-47f9-ac01-5a1c10279434" containerName="revoke-keys"
Nov 01 04:29:19.128675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128660    4882 cpu_manager.go:414] "containerMap: removing container" podUID="53386651-47b5-4947-8b64-5ab4bc64081d" containerName="payload-processing"
Nov 01 04:29:19.128675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128665    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53386651-47b5-4947-8b64-5ab4bc64081d" containerName="payload-processing"
Nov 01 04:29:19.128675 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128677    4882 cpu_manager.go:414] "containerMap: removing container" podUID="fa8f7f58-3d22-463c-a248-8cd08860bd07" containerName="payload-pre-processing"
Nov 01 04:29:19.128856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128682    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8f7f58-3d22-463c-a248-8cd08860bd07" containerName="payload-pre-processing"
Nov 01 04:29:19.128856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128690    4882 cpu_manager.go:414] "containerMap: removing container" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy"
Nov 01 04:29:19.128856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128695    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy"
Nov 01 04:29:19.128856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128780    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed229bec-cfac-47f9-ac01-5a1c10279434" containerName="revoke-keys"
Nov 01 04:29:19.128856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128789    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="53386651-47b5-4947-8b64-5ab4bc64081d" containerName="payload-processing"
Nov 01 04:29:19.128856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128797    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8f7f58-3d22-463c-a248-8cd08860bd07" containerName="payload-pre-processing"
Nov 01 04:29:19.128856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128802    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy"
Nov 01 04:29:19.128856 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.128812    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b18693-924b-492c-9db8-29a52ef1bb12" containerName="istio-proxy"
Nov 01 04:29:19.133631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.133611    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:19.183468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.183439    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:19.183579 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.183492    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dh9b\" (UniqueName: \"kubernetes.io/projected/ccf9a584-91d0-4f8f-af89-7101215305bc-kube-api-access-6dh9b\") pod \"maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:19.183636 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.183617    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:19.227558 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.227529    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:29:19.227686 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.227584    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:29:19.230620 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.230599    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"]
Nov 01 04:29:19.284697 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.284668    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:19.284860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.284806    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:19.284933 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.284859    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dh9b\" (UniqueName: \"kubernetes.io/projected/ccf9a584-91d0-4f8f-af89-7101215305bc-kube-api-access-6dh9b\") pod \"maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:19.285410 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.285390    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:19.287173 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.287147    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:19.341594 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.341563    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dh9b\" (UniqueName: \"kubernetes.io/projected/ccf9a584-91d0-4f8f-af89-7101215305bc-kube-api-access-6dh9b\") pod \"maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:19.446797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.446771    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:19.599460 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.599411    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53386651-47b5-4947-8b64-5ab4bc64081d" path="/var/lib/kubelet/pods/53386651-47b5-4947-8b64-5ab4bc64081d/volumes"
Nov 01 04:29:19.599927 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.599907    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa8f7f58-3d22-463c-a248-8cd08860bd07" path="/var/lib/kubelet/pods/fa8f7f58-3d22-463c-a248-8cd08860bd07/volumes"
Nov 01 04:29:19.815685 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:19.815662    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"]
Nov 01 04:29:19.818072 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:29:19.818031    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf9a584_91d0_4f8f_af89_7101215305bc.slice/crio-036305367fb1d27ffe22483564bce9540474c215d8ed7dba1bd243a495cf0420 WatchSource:0}: Error finding container 036305367fb1d27ffe22483564bce9540474c215d8ed7dba1bd243a495cf0420: Status 404 returned error can't find the container with id 036305367fb1d27ffe22483564bce9540474c215d8ed7dba1bd243a495cf0420
Nov 01 04:29:20.808021 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:20.807934    4882 generic.go:356] "Generic (PLEG): container finished" podID="ccf9a584-91d0-4f8f-af89-7101215305bc" containerID="002a796890ec7617e6344c6861c0d8f35eccc95d5596572d95ce47390cd394d8" exitCode=0
Nov 01 04:29:20.808408 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:20.808024    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4" event={"ID":"ccf9a584-91d0-4f8f-af89-7101215305bc","Type":"ContainerDied","Data":"002a796890ec7617e6344c6861c0d8f35eccc95d5596572d95ce47390cd394d8"}
Nov 01 04:29:20.808408 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:20.808091    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4" event={"ID":"ccf9a584-91d0-4f8f-af89-7101215305bc","Type":"ContainerStarted","Data":"036305367fb1d27ffe22483564bce9540474c215d8ed7dba1bd243a495cf0420"}
Nov 01 04:29:21.984598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:21.984574    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:22.117457 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.117369    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-xks-ca\") pod \"ccf9a584-91d0-4f8f-af89-7101215305bc\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") "
Nov 01 04:29:22.117457 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.117418    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf9a584-91d0-4f8f-af89-7101215305bc-kube-api-access-6dh9b\" (UniqueName: \"kubernetes.io/projected/ccf9a584-91d0-4f8f-af89-7101215305bc-kube-api-access-6dh9b\") pod \"ccf9a584-91d0-4f8f-af89-7101215305bc\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") "
Nov 01 04:29:22.117708 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.117566    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-openshift-ca\") pod \"ccf9a584-91d0-4f8f-af89-7101215305bc\" (UID: \"ccf9a584-91d0-4f8f-af89-7101215305bc\") "
Nov 01 04:29:22.118014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.117983    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-openshift-ca" pod "ccf9a584-91d0-4f8f-af89-7101215305bc" (UID: "ccf9a584-91d0-4f8f-af89-7101215305bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:29:22.118178 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.118137    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:22.119539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.119515    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-xks-ca" pod "ccf9a584-91d0-4f8f-af89-7101215305bc" (UID: "ccf9a584-91d0-4f8f-af89-7101215305bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:29:22.119657 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.119601    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf9a584-91d0-4f8f-af89-7101215305bc-kube-api-access-6dh9b" pod "ccf9a584-91d0-4f8f-af89-7101215305bc" (UID: "ccf9a584-91d0-4f8f-af89-7101215305bc"). InnerVolumeSpecName "kube-api-access-6dh9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:29:22.219289 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.219256    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf9a584-91d0-4f8f-af89-7101215305bc-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:22.219289 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.219286    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dh9b\" (UniqueName: \"kubernetes.io/projected/ccf9a584-91d0-4f8f-af89-7101215305bc-kube-api-access-6dh9b\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:29:22.821706 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.821676    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"
Nov 01 04:29:22.821920 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.821718    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4" event={"ID":"ccf9a584-91d0-4f8f-af89-7101215305bc","Type":"ContainerDied","Data":"036305367fb1d27ffe22483564bce9540474c215d8ed7dba1bd243a495cf0420"}
Nov 01 04:29:22.821920 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:22.821789    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="036305367fb1d27ffe22483564bce9540474c215d8ed7dba1bd243a495cf0420"
Nov 01 04:29:28.725879 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:29:28.725840    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:29:50.548654 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:50.548617    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"]
Nov 01 04:29:50.594937 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:50.594907    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1cdd1d-p7wf4"]
Nov 01 04:29:51.600063 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:51.600025    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf9a584-91d0-4f8f-af89-7101215305bc" path="/var/lib/kubelet/pods/ccf9a584-91d0-4f8f-af89-7101215305bc/volumes"
Nov 01 04:29:56.975872 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:56.975837    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"]
Nov 01 04:29:57.024647 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:57.024610    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-804014-lp9d8"]
Nov 01 04:29:57.035941 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:57.035905    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"]
Nov 01 04:29:57.036135 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:57.036114    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy" containerID="cri-o://eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1" gracePeriod=30
Nov 01 04:29:57.599578 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:29:57.599543    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed229bec-cfac-47f9-ac01-5a1c10279434" path="/var/lib/kubelet/pods/ed229bec-cfac-47f9-ac01-5a1c10279434/volumes"
Nov 01 04:29:58.725844 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:29:58.725794    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:30:00.174905 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.174869    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"]
Nov 01 04:30:00.175417 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.175403    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ccf9a584-91d0-4f8f-af89-7101215305bc" containerName="revoke-keys"
Nov 01 04:30:00.175498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.175418    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf9a584-91d0-4f8f-af89-7101215305bc" containerName="revoke-keys"
Nov 01 04:30:00.175561 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.175551    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf9a584-91d0-4f8f-af89-7101215305bc" containerName="revoke-keys"
Nov 01 04:30:00.180120 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.180099    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"]
Nov 01 04:30:00.180256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.180219    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"
Nov 01 04:30:00.183913 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.183894    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"]
Nov 01 04:30:00.184033 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.184013    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"
Nov 01 04:30:00.187545 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.187525    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"
Nov 01 04:30:00.220767 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.220738    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"]
Nov 01 04:30:00.224309 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.224292    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"]
Nov 01 04:30:00.224421 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.224405    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"
Nov 01 04:30:00.227960 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.227945    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"]
Nov 01 04:30:00.228074 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.228055    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"
Nov 01 04:30:00.228605 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.228588    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:30:00.232614 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.232591    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"
Nov 01 04:30:00.240664 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.240638    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw7kk\" (UniqueName: \"kubernetes.io/projected/5b73bbcd-28bf-401f-97f4-301dd26d4c46-kube-api-access-cw7kk\") pod \"maas-api-key-cleanup-29891790-7z4b2\" (UID: \"5b73bbcd-28bf-401f-97f4-301dd26d4c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"
Nov 01 04:30:00.240762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.240742    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4f2\" (UniqueName: \"kubernetes.io/projected/e5771967-299c-4e10-bd92-2637c3ec175b-kube-api-access-9c4f2\") pod \"maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w\" (UID: \"e5771967-299c-4e10-bd92-2637c3ec175b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"
Nov 01 04:30:00.240874 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.240848    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1a2d6a-5672-4106-9332-59330be63eec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h\" (UID: \"5c1a2d6a-5672-4106-9332-59330be63eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"
Nov 01 04:30:00.240942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.240924    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5771967-299c-4e10-bd92-2637c3ec175b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w\" (UID: \"e5771967-299c-4e10-bd92-2637c3ec175b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"
Nov 01 04:30:00.241049 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.241033    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpb4t\" (UniqueName: \"kubernetes.io/projected/5c1a2d6a-5672-4106-9332-59330be63eec-kube-api-access-zpb4t\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h\" (UID: \"5c1a2d6a-5672-4106-9332-59330be63eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"
Nov 01 04:30:00.241100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.241077    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b73bbcd-28bf-401f-97f4-301dd26d4c46-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891790-7z4b2\" (UID: \"5b73bbcd-28bf-401f-97f4-301dd26d4c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"
Nov 01 04:30:00.276161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.276135    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"]
Nov 01 04:30:00.279371 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.279341    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"]
Nov 01 04:30:00.326755 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.326708    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"]
Nov 01 04:30:00.330750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.330716    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"]
Nov 01 04:30:00.334976 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.334949    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"]
Nov 01 04:30:00.342419 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.342389    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1a2d6a-5672-4106-9332-59330be63eec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h\" (UID: \"5c1a2d6a-5672-4106-9332-59330be63eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"
Nov 01 04:30:00.342573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.342494    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5771967-299c-4e10-bd92-2637c3ec175b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w\" (UID: \"e5771967-299c-4e10-bd92-2637c3ec175b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"
Nov 01 04:30:00.342573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.342530    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr\" (UID: \"ba6a23a3-2d0f-4083-a42e-3067dbe62be7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"
Nov 01 04:30:00.342705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.342620    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4rgt\" (UniqueName: \"kubernetes.io/projected/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-kube-api-access-p4rgt\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr\" (UID: \"ba6a23a3-2d0f-4083-a42e-3067dbe62be7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"
Nov 01 04:30:00.342773 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.342720    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpb4t\" (UniqueName: \"kubernetes.io/projected/5c1a2d6a-5672-4106-9332-59330be63eec-kube-api-access-zpb4t\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h\" (UID: \"5c1a2d6a-5672-4106-9332-59330be63eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"
Nov 01 04:30:00.342834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.342770    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx\" (UID: \"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"
Nov 01 04:30:00.342834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.342827    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b73bbcd-28bf-401f-97f4-301dd26d4c46-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891790-7z4b2\" (UID: \"5b73bbcd-28bf-401f-97f4-301dd26d4c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"
Nov 01 04:30:00.342962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.342863    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7kk\" (UniqueName: \"kubernetes.io/projected/5b73bbcd-28bf-401f-97f4-301dd26d4c46-kube-api-access-cw7kk\") pod \"maas-api-key-cleanup-29891790-7z4b2\" (UID: \"5b73bbcd-28bf-401f-97f4-301dd26d4c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"
Nov 01 04:30:00.342962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.342947    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfpm\" (UniqueName: \"kubernetes.io/projected/53d9aefe-476b-49e5-8b67-7f8cb2768be8-kube-api-access-vnfpm\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m\" (UID: \"53d9aefe-476b-49e5-8b67-7f8cb2768be8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"
Nov 01 04:30:00.343091 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.342998    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4f2\" (UniqueName: \"kubernetes.io/projected/e5771967-299c-4e10-bd92-2637c3ec175b-kube-api-access-9c4f2\") pod \"maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w\" (UID: \"e5771967-299c-4e10-bd92-2637c3ec175b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"
Nov 01 04:30:00.343091 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.343024    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d9aefe-476b-49e5-8b67-7f8cb2768be8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m\" (UID: \"53d9aefe-476b-49e5-8b67-7f8cb2768be8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"
Nov 01 04:30:00.343091 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.343053    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtcmg\" (UniqueName: \"kubernetes.io/projected/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-kube-api-access-dtcmg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx\" (UID: \"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"
Nov 01 04:30:00.343264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.343204    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1a2d6a-5672-4106-9332-59330be63eec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h\" (UID: \"5c1a2d6a-5672-4106-9332-59330be63eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"
Nov 01 04:30:00.343264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.343232    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5771967-299c-4e10-bd92-2637c3ec175b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w\" (UID: \"e5771967-299c-4e10-bd92-2637c3ec175b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"
Nov 01 04:30:00.343870 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.343849    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b73bbcd-28bf-401f-97f4-301dd26d4c46-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891790-7z4b2\" (UID: \"5b73bbcd-28bf-401f-97f4-301dd26d4c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"
Nov 01 04:30:00.395047 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.395005    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"]
Nov 01 04:30:00.404950 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.404497    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7kk\" (UniqueName: \"kubernetes.io/projected/5b73bbcd-28bf-401f-97f4-301dd26d4c46-kube-api-access-cw7kk\") pod \"maas-api-key-cleanup-29891790-7z4b2\" (UID: \"5b73bbcd-28bf-401f-97f4-301dd26d4c46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"
Nov 01 04:30:00.404950 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.404598    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpb4t\" (UniqueName: \"kubernetes.io/projected/5c1a2d6a-5672-4106-9332-59330be63eec-kube-api-access-zpb4t\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h\" (UID: \"5c1a2d6a-5672-4106-9332-59330be63eec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"
Nov 01 04:30:00.404950 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.404901    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4f2\" (UniqueName: \"kubernetes.io/projected/e5771967-299c-4e10-bd92-2637c3ec175b-kube-api-access-9c4f2\") pod \"maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w\" (UID: \"e5771967-299c-4e10-bd92-2637c3ec175b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"
Nov 01 04:30:00.443946 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.443913    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4rgt\" (UniqueName: \"kubernetes.io/projected/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-kube-api-access-p4rgt\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr\" (UID: \"ba6a23a3-2d0f-4083-a42e-3067dbe62be7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"
Nov 01 04:30:00.444120 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.443964    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx\" (UID: \"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"
Nov 01 04:30:00.444120 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.444034    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfpm\" (UniqueName: \"kubernetes.io/projected/53d9aefe-476b-49e5-8b67-7f8cb2768be8-kube-api-access-vnfpm\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m\" (UID: \"53d9aefe-476b-49e5-8b67-7f8cb2768be8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"
Nov 01 04:30:00.444120 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.444060    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d9aefe-476b-49e5-8b67-7f8cb2768be8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m\" (UID: \"53d9aefe-476b-49e5-8b67-7f8cb2768be8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"
Nov 01 04:30:00.444120 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.444078    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtcmg\" (UniqueName: \"kubernetes.io/projected/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-kube-api-access-dtcmg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx\" (UID: \"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"
Nov 01 04:30:00.444383 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.444122    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr\" (UID: \"ba6a23a3-2d0f-4083-a42e-3067dbe62be7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"
Nov 01 04:30:00.444770 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.444748    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr\" (UID: \"ba6a23a3-2d0f-4083-a42e-3067dbe62be7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"
Nov 01 04:30:00.444860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.444832    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d9aefe-476b-49e5-8b67-7f8cb2768be8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m\" (UID: \"53d9aefe-476b-49e5-8b67-7f8cb2768be8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"
Nov 01 04:30:00.444911 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.444830    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx\" (UID: \"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"
Nov 01 04:30:00.494354 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.494315    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"
Nov 01 04:30:00.501290 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.501260    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtcmg\" (UniqueName: \"kubernetes.io/projected/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-kube-api-access-dtcmg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx\" (UID: \"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"
Nov 01 04:30:00.505755 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.505730    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"
Nov 01 04:30:00.506163 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.506143    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfpm\" (UniqueName: \"kubernetes.io/projected/53d9aefe-476b-49e5-8b67-7f8cb2768be8-kube-api-access-vnfpm\") pod \"maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m\" (UID: \"53d9aefe-476b-49e5-8b67-7f8cb2768be8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"
Nov 01 04:30:00.506253 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.506209    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4rgt\" (UniqueName: \"kubernetes.io/projected/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-kube-api-access-p4rgt\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr\" (UID: \"ba6a23a3-2d0f-4083-a42e-3067dbe62be7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"
Nov 01 04:30:00.508546 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.508518    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"
Nov 01 04:30:00.548691 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.548654    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"
Nov 01 04:30:00.555257 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.555163    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"
Nov 01 04:30:00.567667 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.563258    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"
Nov 01 04:30:00.887539 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.887509    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"]
Nov 01 04:30:00.891259 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:30:00.891224    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5771967_299c_4e10_bd92_2637c3ec175b.slice/crio-5db270c0cbe672bf0a0cf7caa33536446f708c359fefceb2bc097e2455a78314 WatchSource:0}: Error finding container 5db270c0cbe672bf0a0cf7caa33536446f708c359fefceb2bc097e2455a78314: Status 404 returned error can't find the container with id 5db270c0cbe672bf0a0cf7caa33536446f708c359fefceb2bc097e2455a78314
Nov 01 04:30:00.933376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.933353    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"]
Nov 01 04:30:00.935082 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:30:00.935057    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1a2d6a_5672_4106_9332_59330be63eec.slice/crio-11b62e356be1ca5b63b22f9ff352127c8bb682d5d1c320498f8a1a8dceba0951 WatchSource:0}: Error finding container 11b62e356be1ca5b63b22f9ff352127c8bb682d5d1c320498f8a1a8dceba0951: Status 404 returned error can't find the container with id 11b62e356be1ca5b63b22f9ff352127c8bb682d5d1c320498f8a1a8dceba0951
Nov 01 04:30:00.936953 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.936934    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"]
Nov 01 04:30:00.938419 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:30:00.938384    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b73bbcd_28bf_401f_97f4_301dd26d4c46.slice/crio-f73778a19417dbfa95f5d3f0d2b907eb0538e5b07c91abd1aed86dc2eb2dd1aa WatchSource:0}: Error finding container f73778a19417dbfa95f5d3f0d2b907eb0538e5b07c91abd1aed86dc2eb2dd1aa: Status 404 returned error can't find the container with id f73778a19417dbfa95f5d3f0d2b907eb0538e5b07c91abd1aed86dc2eb2dd1aa
Nov 01 04:30:00.984764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.984738    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"]
Nov 01 04:30:00.987197 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:30:00.987172    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d9aefe_476b_49e5_8b67_7f8cb2768be8.slice/crio-984150c90d82b32fc8da5e3d60b7d1c2e472d5bb8f4e3c8db71a53d4b7131b91 WatchSource:0}: Error finding container 984150c90d82b32fc8da5e3d60b7d1c2e472d5bb8f4e3c8db71a53d4b7131b91: Status 404 returned error can't find the container with id 984150c90d82b32fc8da5e3d60b7d1c2e472d5bb8f4e3c8db71a53d4b7131b91
Nov 01 04:30:00.987873 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:30:00.987826    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6a23a3_2d0f_4083_a42e_3067dbe62be7.slice/crio-18e66932345266af0f23b1ba0f07b0f350ca59bbec26923236baae240cbd2bc6 WatchSource:0}: Error finding container 18e66932345266af0f23b1ba0f07b0f350ca59bbec26923236baae240cbd2bc6: Status 404 returned error can't find the container with id 18e66932345266af0f23b1ba0f07b0f350ca59bbec26923236baae240cbd2bc6
Nov 01 04:30:00.988137 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.988086    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"]
Nov 01 04:30:00.988797 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:30:00.988773    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc456f08_2b1d_4434_ac72_bc5eb9a8ac8c.slice/crio-1f5a61e422297cbaf8d9b6477fae8499b6b6c0adecaf8490d55fb657a65a148b WatchSource:0}: Error finding container 1f5a61e422297cbaf8d9b6477fae8499b6b6c0adecaf8490d55fb657a65a148b: Status 404 returned error can't find the container with id 1f5a61e422297cbaf8d9b6477fae8499b6b6c0adecaf8490d55fb657a65a148b
Nov 01 04:30:00.991587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:00.991567    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"]
Nov 01 04:30:01.075629 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:01.075599    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2" event={"ID":"5b73bbcd-28bf-401f-97f4-301dd26d4c46","Type":"ContainerStarted","Data":"f73778a19417dbfa95f5d3f0d2b907eb0538e5b07c91abd1aed86dc2eb2dd1aa"}
Nov 01 04:30:01.076818 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:01.076795    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w" event={"ID":"e5771967-299c-4e10-bd92-2637c3ec175b","Type":"ContainerStarted","Data":"5db270c0cbe672bf0a0cf7caa33536446f708c359fefceb2bc097e2455a78314"}
Nov 01 04:30:01.078135 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:01.078108    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" event={"ID":"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c","Type":"ContainerStarted","Data":"1f5a61e422297cbaf8d9b6477fae8499b6b6c0adecaf8490d55fb657a65a148b"}
Nov 01 04:30:01.079287 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:01.079269    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m" event={"ID":"53d9aefe-476b-49e5-8b67-7f8cb2768be8","Type":"ContainerStarted","Data":"984150c90d82b32fc8da5e3d60b7d1c2e472d5bb8f4e3c8db71a53d4b7131b91"}
Nov 01 04:30:01.080419 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:01.080399    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" event={"ID":"5c1a2d6a-5672-4106-9332-59330be63eec","Type":"ContainerStarted","Data":"11b62e356be1ca5b63b22f9ff352127c8bb682d5d1c320498f8a1a8dceba0951"}
Nov 01 04:30:01.081551 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:01.081530    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" event={"ID":"ba6a23a3-2d0f-4083-a42e-3067dbe62be7","Type":"ContainerStarted","Data":"18e66932345266af0f23b1ba0f07b0f350ca59bbec26923236baae240cbd2bc6"}
Nov 01 04:30:02.089860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.089824    4882 generic.go:356] "Generic (PLEG): container finished" podID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerID="3f58ea934596891a9538f9d357ab5ccbb2d45df3f433cc45bdde1c99b1362405" exitCode=60
Nov 01 04:30:02.090284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.089918    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2" event={"ID":"5b73bbcd-28bf-401f-97f4-301dd26d4c46","Type":"ContainerDied","Data":"3f58ea934596891a9538f9d357ab5ccbb2d45df3f433cc45bdde1c99b1362405"}
Nov 01 04:30:02.090284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.090192    4882 scope.go:122] "RemoveContainer" containerID="3f58ea934596891a9538f9d357ab5ccbb2d45df3f433cc45bdde1c99b1362405"
Nov 01 04:30:02.091871 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.091769    4882 generic.go:356] "Generic (PLEG): container finished" podID="e5771967-299c-4e10-bd92-2637c3ec175b" containerID="be1fd8933e45d44312fcfae3a67586610547f92ef508314fe0dee4b562ff03ce" exitCode=60
Nov 01 04:30:02.091871 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.091807    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w" event={"ID":"e5771967-299c-4e10-bd92-2637c3ec175b","Type":"ContainerDied","Data":"be1fd8933e45d44312fcfae3a67586610547f92ef508314fe0dee4b562ff03ce"}
Nov 01 04:30:02.092117 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.092097    4882 scope.go:122] "RemoveContainer" containerID="be1fd8933e45d44312fcfae3a67586610547f92ef508314fe0dee4b562ff03ce"
Nov 01 04:30:02.100159 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.099299    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" event={"ID":"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c","Type":"ContainerStarted","Data":"8571ae7a5c5ded72f4ed6397806dd62686b719b0fe20457145930254eef26bc3"}
Nov 01 04:30:02.103533 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.103488    4882 generic.go:356] "Generic (PLEG): container finished" podID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerID="10519ea49cd7d348ff0d04cd8585e8385577c28d904fdda523763944e7f6f7e9" exitCode=60
Nov 01 04:30:02.103808 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.103626    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m" event={"ID":"53d9aefe-476b-49e5-8b67-7f8cb2768be8","Type":"ContainerDied","Data":"10519ea49cd7d348ff0d04cd8585e8385577c28d904fdda523763944e7f6f7e9"}
Nov 01 04:30:02.104244 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.104221    4882 scope.go:122] "RemoveContainer" containerID="10519ea49cd7d348ff0d04cd8585e8385577c28d904fdda523763944e7f6f7e9"
Nov 01 04:30:02.109071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.106526    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" event={"ID":"5c1a2d6a-5672-4106-9332-59330be63eec","Type":"ContainerStarted","Data":"200dc5a275932be009823faf7e0d11464e5e79f9bbfe736e14a030b466ffb53a"}
Nov 01 04:30:02.109743 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.109717    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" event={"ID":"ba6a23a3-2d0f-4083-a42e-3067dbe62be7","Type":"ContainerStarted","Data":"c7a43b056e5b943f93c1951c62edaeacb09c037c3b892a72eca8e567d40838f5"}
Nov 01 04:30:02.305903 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.305850    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" podStartSLOduration=2.305835921 podStartE2EDuration="2.305835921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:30:02.302524377 +0000 UTC m=+2870.864557394" watchObservedRunningTime="2026-11-01 04:30:02.305835921 +0000 UTC m=+2870.867868937"
Nov 01 04:30:02.333219 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.333189    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:30:02.410921 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.410866    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" podStartSLOduration=2.410852665 podStartE2EDuration="2.410852665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:30:02.408640672 +0000 UTC m=+2870.970673688" watchObservedRunningTime="2026-11-01 04:30:02.410852665 +0000 UTC m=+2870.972885679"
Nov 01 04:30:02.473454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.473228    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-kube-api-access-qq5mh\" (UniqueName: \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-kube-api-access-qq5mh\") pod \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") "
Nov 01 04:30:02.473454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.473290    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-socket\") pod \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") "
Nov 01 04:30:02.473454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.473404    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-token\" (UniqueName: \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-token\") pod \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") "
Nov 01 04:30:02.473759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.473483    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-credential-socket\") pod \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") "
Nov 01 04:30:02.473759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.473565    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-envoy\") pod \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") "
Nov 01 04:30:02.473759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.473596    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-certs\") pod \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") "
Nov 01 04:30:02.473759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.473658    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-podinfo\") pod \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") "
Nov 01 04:30:02.473759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.473729    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istiod-ca-cert\") pod \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") "
Nov 01 04:30:02.473759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.473758    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-data\") pod \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\" (UID: \"9390f9be-7ca3-4bf4-9da5-65cdc7daba97\") "
Nov 01 04:30:02.475050 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.474968    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-data" pod "9390f9be-7ca3-4bf4-9da5-65cdc7daba97" (UID: "9390f9be-7ca3-4bf4-9da5-65cdc7daba97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:02.475582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.475250    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-socket" pod "9390f9be-7ca3-4bf4-9da5-65cdc7daba97" (UID: "9390f9be-7ca3-4bf4-9da5-65cdc7daba97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:02.475582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.475536    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istiod-ca-cert" pod "9390f9be-7ca3-4bf4-9da5-65cdc7daba97" (UID: "9390f9be-7ca3-4bf4-9da5-65cdc7daba97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:02.476368 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.476281    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-credential-socket" pod "9390f9be-7ca3-4bf4-9da5-65cdc7daba97" (UID: "9390f9be-7ca3-4bf4-9da5-65cdc7daba97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:02.476736 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.476711    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-certs" pod "9390f9be-7ca3-4bf4-9da5-65cdc7daba97" (UID: "9390f9be-7ca3-4bf4-9da5-65cdc7daba97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:02.477176 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.477153    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-kube-api-access-qq5mh" pod "9390f9be-7ca3-4bf4-9da5-65cdc7daba97" (UID: "9390f9be-7ca3-4bf4-9da5-65cdc7daba97"). InnerVolumeSpecName "kube-api-access-qq5mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:02.477419 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.477361    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-envoy" pod "9390f9be-7ca3-4bf4-9da5-65cdc7daba97" (UID: "9390f9be-7ca3-4bf4-9da5-65cdc7daba97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:02.477682 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.477657    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-podinfo" pod "9390f9be-7ca3-4bf4-9da5-65cdc7daba97" (UID: "9390f9be-7ca3-4bf4-9da5-65cdc7daba97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:30:02.478266 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.478247    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-token" pod "9390f9be-7ca3-4bf4-9da5-65cdc7daba97" (UID: "9390f9be-7ca3-4bf4-9da5-65cdc7daba97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:02.575631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.575607    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:02.575774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.575633    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:02.575774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.575647    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq5mh\" (UniqueName: \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-kube-api-access-qq5mh\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:02.575774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.575661    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:02.575774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.575673    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:02.575774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.575687    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:02.575774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.575699    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:02.575774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.575712    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:02.575774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.575724    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9390f9be-7ca3-4bf4-9da5-65cdc7daba97-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:02.729554 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:02.729453    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" podStartSLOduration=2.729440278 podStartE2EDuration="2.729440278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:30:02.726334256 +0000 UTC m=+2871.288367270" watchObservedRunningTime="2026-11-01 04:30:02.729440278 +0000 UTC m=+2871.291473286"
Nov 01 04:30:03.117502 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.117461    4882 generic.go:356] "Generic (PLEG): container finished" podID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerID="c7a43b056e5b943f93c1951c62edaeacb09c037c3b892a72eca8e567d40838f5" exitCode=60
Nov 01 04:30:03.117934 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.117522    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" event={"ID":"ba6a23a3-2d0f-4083-a42e-3067dbe62be7","Type":"ContainerDied","Data":"c7a43b056e5b943f93c1951c62edaeacb09c037c3b892a72eca8e567d40838f5"}
Nov 01 04:30:03.117934 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.117806    4882 scope.go:122] "RemoveContainer" containerID="c7a43b056e5b943f93c1951c62edaeacb09c037c3b892a72eca8e567d40838f5"
Nov 01 04:30:03.119850 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.119827    4882 generic.go:356] "Generic (PLEG): container finished" podID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerID="482dbf3061a2f1b2e4f08868e7d50f837094bdcdef16628d57a4e5dc184045df" exitCode=60
Nov 01 04:30:03.119954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.119856    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2" event={"ID":"5b73bbcd-28bf-401f-97f4-301dd26d4c46","Type":"ContainerDied","Data":"482dbf3061a2f1b2e4f08868e7d50f837094bdcdef16628d57a4e5dc184045df"}
Nov 01 04:30:03.119954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.119913    4882 scope.go:122] "RemoveContainer" containerID="3f58ea934596891a9538f9d357ab5ccbb2d45df3f433cc45bdde1c99b1362405"
Nov 01 04:30:03.120161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.120144    4882 scope.go:122] "RemoveContainer" containerID="482dbf3061a2f1b2e4f08868e7d50f837094bdcdef16628d57a4e5dc184045df"
Nov 01 04:30:03.121150 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:03.120809    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891790-7z4b2_odh-ai-gateway-infra(5b73bbcd-28bf-401f-97f4-301dd26d4c46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46"
Nov 01 04:30:03.122039 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.122015    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w" event={"ID":"e5771967-299c-4e10-bd92-2637c3ec175b","Type":"ContainerStarted","Data":"aba31400f6c0343b3fe91b68f43a68beea4bf9c80c3606b9103b23e0d34be750"}
Nov 01 04:30:03.123703 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.123682    4882 generic.go:356] "Generic (PLEG): container finished" podID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerID="8571ae7a5c5ded72f4ed6397806dd62686b719b0fe20457145930254eef26bc3" exitCode=60
Nov 01 04:30:03.123788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.123751    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" event={"ID":"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c","Type":"ContainerDied","Data":"8571ae7a5c5ded72f4ed6397806dd62686b719b0fe20457145930254eef26bc3"}
Nov 01 04:30:03.123955 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.123941    4882 scope.go:122] "RemoveContainer" containerID="8571ae7a5c5ded72f4ed6397806dd62686b719b0fe20457145930254eef26bc3"
Nov 01 04:30:03.125921 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.125899    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m" event={"ID":"53d9aefe-476b-49e5-8b67-7f8cb2768be8","Type":"ContainerStarted","Data":"be763ab98747aa19180ee9707cc5bc3229be4187dd4f9836f3ae65ed1a5c685a"}
Nov 01 04:30:03.132410 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.132383    4882 generic.go:356] "Generic (PLEG): container finished" podID="5c1a2d6a-5672-4106-9332-59330be63eec" containerID="200dc5a275932be009823faf7e0d11464e5e79f9bbfe736e14a030b466ffb53a" exitCode=60
Nov 01 04:30:03.132555 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.132513    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" event={"ID":"5c1a2d6a-5672-4106-9332-59330be63eec","Type":"ContainerDied","Data":"200dc5a275932be009823faf7e0d11464e5e79f9bbfe736e14a030b466ffb53a"}
Nov 01 04:30:03.132834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.132795    4882 scope.go:122] "RemoveContainer" containerID="200dc5a275932be009823faf7e0d11464e5e79f9bbfe736e14a030b466ffb53a"
Nov 01 04:30:03.134717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.134695    4882 generic.go:356] "Generic (PLEG): container finished" podID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerID="eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1" exitCode=0
Nov 01 04:30:03.134834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.134743    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"
Nov 01 04:30:03.134834 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.134745    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" event={"ID":"9390f9be-7ca3-4bf4-9da5-65cdc7daba97","Type":"ContainerDied","Data":"eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1"}
Nov 01 04:30:03.134955 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.134871    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc" event={"ID":"9390f9be-7ca3-4bf4-9da5-65cdc7daba97","Type":"ContainerDied","Data":"501c36fdf7922d917adcb3f5c8fbe6b485080623c416bb563997f0c2abe51c97"}
Nov 01 04:30:03.137612 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.137592    4882 scope.go:122] "RemoveContainer" containerID="eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1"
Nov 01 04:30:03.159365 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.159347    4882 scope.go:122] "RemoveContainer" containerID="b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9"
Nov 01 04:30:03.332447 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.332401    4882 scope.go:122] "RemoveContainer" containerID="eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1"
Nov 01 04:30:03.332865 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:03.332840    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1\": container with ID starting with eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1 not found: ID does not exist" containerID="eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1"
Nov 01 04:30:03.332939 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.332875    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1"} err="failed to get container status \"eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1\": rpc error: code = NotFound desc = could not find container \"eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1\": container with ID starting with eb5501fd34acbd717aea37dd2a8f94beb3a6e69b96faec445e833005524cf1c1 not found: ID does not exist"
Nov 01 04:30:03.332939 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.332896    4882 scope.go:122] "RemoveContainer" containerID="b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9"
Nov 01 04:30:03.333238 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:03.333223    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9\": container with ID starting with b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9 not found: ID does not exist" containerID="b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9"
Nov 01 04:30:03.333282 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.333242    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9"} err="failed to get container status \"b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9\": rpc error: code = NotFound desc = could not find container \"b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9\": container with ID starting with b9c5935ce6e3d807a66f473c9ce4e61b873000ff8a7cfea115309d5690f1b1e9 not found: ID does not exist"
Nov 01 04:30:03.406561 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.406530    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"]
Nov 01 04:30:03.407089 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.407074    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy"
Nov 01 04:30:03.407139 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.407090    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy"
Nov 01 04:30:03.407139 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.407120    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy"
Nov 01 04:30:03.407139 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.407125    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy"
Nov 01 04:30:03.407245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.407207    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy"
Nov 01 04:30:03.407245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.407222    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" containerName="istio-proxy"
Nov 01 04:30:03.412215 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.412193    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:03.440454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.440378    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w" podStartSLOduration=3.4403643280000002 podStartE2EDuration="3.440364328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:30:03.438093591 +0000 UTC m=+2872.000126608" watchObservedRunningTime="2026-11-01 04:30:03.440364328 +0000 UTC m=+2872.002397345"
Nov 01 04:30:03.460615 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.460588    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:30:03.467774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.467746    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"]
Nov 01 04:30:03.495763 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.495732    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:03.495907 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.495835    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:03.495998 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.495907    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rck65\" (UniqueName: \"kubernetes.io/projected/8bb84c12-e9cf-4428-b367-e617b91f4c03-kube-api-access-rck65\") pod \"maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:03.596873 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.596756    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rck65\" (UniqueName: \"kubernetes.io/projected/8bb84c12-e9cf-4428-b367-e617b91f4c03-kube-api-access-rck65\") pod \"maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:03.597054 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.596946    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:03.597054 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.596999    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:03.598458 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.597867    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:03.602829 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.602804    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:03.653931 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.653898    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rck65\" (UniqueName: \"kubernetes.io/projected/8bb84c12-e9cf-4428-b367-e617b91f4c03-kube-api-access-rck65\") pod \"maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:03.808404 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.808347    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m" podStartSLOduration=3.808333736 podStartE2EDuration="3.808333736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:30:03.806106127 +0000 UTC m=+2872.368139144" watchObservedRunningTime="2026-11-01 04:30:03.808333736 +0000 UTC m=+2872.370366754"
Nov 01 04:30:03.813239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:03.813211    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:04.017565 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.017535    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"]
Nov 01 04:30:04.021854 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.021828    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-1cdd1d-openshift-default-784c85765c-tphpc"]
Nov 01 04:30:04.144844 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.144762    4882 scope.go:122] "RemoveContainer" containerID="482dbf3061a2f1b2e4f08868e7d50f837094bdcdef16628d57a4e5dc184045df"
Nov 01 04:30:04.145279 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:04.145121    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891790-7z4b2_odh-ai-gateway-infra(5b73bbcd-28bf-401f-97f4-301dd26d4c46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46"
Nov 01 04:30:04.146676 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.146647    4882 generic.go:356] "Generic (PLEG): container finished" podID="e5771967-299c-4e10-bd92-2637c3ec175b" containerID="aba31400f6c0343b3fe91b68f43a68beea4bf9c80c3606b9103b23e0d34be750" exitCode=60
Nov 01 04:30:04.146801 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.146739    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w" event={"ID":"e5771967-299c-4e10-bd92-2637c3ec175b","Type":"ContainerDied","Data":"aba31400f6c0343b3fe91b68f43a68beea4bf9c80c3606b9103b23e0d34be750"}
Nov 01 04:30:04.146872 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.146834    4882 scope.go:122] "RemoveContainer" containerID="be1fd8933e45d44312fcfae3a67586610547f92ef508314fe0dee4b562ff03ce"
Nov 01 04:30:04.146960 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.146943    4882 scope.go:122] "RemoveContainer" containerID="aba31400f6c0343b3fe91b68f43a68beea4bf9c80c3606b9103b23e0d34be750"
Nov 01 04:30:04.147233 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:04.147207    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w_odh-ai-gateway-infra(e5771967-299c-4e10-bd92-2637c3ec175b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w" podUID="e5771967-299c-4e10-bd92-2637c3ec175b"
Nov 01 04:30:04.148903 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.148883    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" event={"ID":"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c","Type":"ContainerStarted","Data":"a3508c48c2b6bfd7c5f0608c17a86b4d5220bfde221f63198c38819ff662a30b"}
Nov 01 04:30:04.151329 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.151298    4882 generic.go:356] "Generic (PLEG): container finished" podID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerID="be763ab98747aa19180ee9707cc5bc3229be4187dd4f9836f3ae65ed1a5c685a" exitCode=60
Nov 01 04:30:04.151597 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.151391    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m" event={"ID":"53d9aefe-476b-49e5-8b67-7f8cb2768be8","Type":"ContainerDied","Data":"be763ab98747aa19180ee9707cc5bc3229be4187dd4f9836f3ae65ed1a5c685a"}
Nov 01 04:30:04.151814 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.151688    4882 scope.go:122] "RemoveContainer" containerID="be763ab98747aa19180ee9707cc5bc3229be4187dd4f9836f3ae65ed1a5c685a"
Nov 01 04:30:04.154418 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:04.152158    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m_odh-ai-gateway-infra(53d9aefe-476b-49e5-8b67-7f8cb2768be8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8"
Nov 01 04:30:04.158223 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.158192    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" event={"ID":"5c1a2d6a-5672-4106-9332-59330be63eec","Type":"ContainerStarted","Data":"f8ccb1f40bbad9cb079ba838cfe16de54befa657987e6cca0670a0e1e8842c12"}
Nov 01 04:30:04.164790 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.164759    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" event={"ID":"ba6a23a3-2d0f-4083-a42e-3067dbe62be7","Type":"ContainerStarted","Data":"230890e85439c1b00a321830d957e4e84fec59b776e5699e90bb765502293c43"}
Nov 01 04:30:04.169833 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.169798    4882 scope.go:122] "RemoveContainer" containerID="10519ea49cd7d348ff0d04cd8585e8385577c28d904fdda523763944e7f6f7e9"
Nov 01 04:30:04.195641 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:30:04.195612    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb84c12_e9cf_4428_b367_e617b91f4c03.slice/crio-faf4688e5bd2cdd25b4d9b2eb1ca0325854e719b8903c98a1d5dde34c2d0e057 WatchSource:0}: Error finding container faf4688e5bd2cdd25b4d9b2eb1ca0325854e719b8903c98a1d5dde34c2d0e057: Status 404 returned error can't find the container with id faf4688e5bd2cdd25b4d9b2eb1ca0325854e719b8903c98a1d5dde34c2d0e057
Nov 01 04:30:04.195859 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.195835    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"]
Nov 01 04:30:04.321193 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.320226    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"]
Nov 01 04:30:04.321193 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:04.321120    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy" containerID="cri-o://f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe" gracePeriod=30
Nov 01 04:30:05.181386 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.181342    4882 generic.go:356] "Generic (PLEG): container finished" podID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerID="230890e85439c1b00a321830d957e4e84fec59b776e5699e90bb765502293c43" exitCode=60
Nov 01 04:30:05.181852 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.181437    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" event={"ID":"ba6a23a3-2d0f-4083-a42e-3067dbe62be7","Type":"ContainerDied","Data":"230890e85439c1b00a321830d957e4e84fec59b776e5699e90bb765502293c43"}
Nov 01 04:30:05.181852 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.181548    4882 scope.go:122] "RemoveContainer" containerID="c7a43b056e5b943f93c1951c62edaeacb09c037c3b892a72eca8e567d40838f5"
Nov 01 04:30:05.181852 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.181787    4882 scope.go:122] "RemoveContainer" containerID="230890e85439c1b00a321830d957e4e84fec59b776e5699e90bb765502293c43"
Nov 01 04:30:05.182109 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:05.182085    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr_odh-ai-gateway-infra(ba6a23a3-2d0f-4083-a42e-3067dbe62be7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7"
Nov 01 04:30:05.183638 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.183623    4882 scope.go:122] "RemoveContainer" containerID="aba31400f6c0343b3fe91b68f43a68beea4bf9c80c3606b9103b23e0d34be750"
Nov 01 04:30:05.183874 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:05.183851    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w_odh-ai-gateway-infra(e5771967-299c-4e10-bd92-2637c3ec175b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w" podUID="e5771967-299c-4e10-bd92-2637c3ec175b"
Nov 01 04:30:05.185446 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.185413    4882 generic.go:356] "Generic (PLEG): container finished" podID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerID="a3508c48c2b6bfd7c5f0608c17a86b4d5220bfde221f63198c38819ff662a30b" exitCode=60
Nov 01 04:30:05.185553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.185509    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" event={"ID":"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c","Type":"ContainerDied","Data":"a3508c48c2b6bfd7c5f0608c17a86b4d5220bfde221f63198c38819ff662a30b"}
Nov 01 04:30:05.185704 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.185689    4882 scope.go:122] "RemoveContainer" containerID="a3508c48c2b6bfd7c5f0608c17a86b4d5220bfde221f63198c38819ff662a30b"
Nov 01 04:30:05.185951 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:05.185929    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx_odh-ai-gateway-infra(bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c"
Nov 01 04:30:05.187081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.187061    4882 generic.go:356] "Generic (PLEG): container finished" podID="8bb84c12-e9cf-4428-b367-e617b91f4c03" containerID="164afddb403bea809f7ca291e8e86526d45d1bde259dcec702fde90a9ba1a56f" exitCode=0
Nov 01 04:30:05.187186 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.187127    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2" event={"ID":"8bb84c12-e9cf-4428-b367-e617b91f4c03","Type":"ContainerDied","Data":"164afddb403bea809f7ca291e8e86526d45d1bde259dcec702fde90a9ba1a56f"}
Nov 01 04:30:05.187186 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.187164    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2" event={"ID":"8bb84c12-e9cf-4428-b367-e617b91f4c03","Type":"ContainerStarted","Data":"faf4688e5bd2cdd25b4d9b2eb1ca0325854e719b8903c98a1d5dde34c2d0e057"}
Nov 01 04:30:05.188779 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.188762    4882 scope.go:122] "RemoveContainer" containerID="be763ab98747aa19180ee9707cc5bc3229be4187dd4f9836f3ae65ed1a5c685a"
Nov 01 04:30:05.188985 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:05.188962    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m_odh-ai-gateway-infra(53d9aefe-476b-49e5-8b67-7f8cb2768be8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8"
Nov 01 04:30:05.190305 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.190288    4882 generic.go:356] "Generic (PLEG): container finished" podID="5c1a2d6a-5672-4106-9332-59330be63eec" containerID="f8ccb1f40bbad9cb079ba838cfe16de54befa657987e6cca0670a0e1e8842c12" exitCode=60
Nov 01 04:30:05.190385 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.190339    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" event={"ID":"5c1a2d6a-5672-4106-9332-59330be63eec","Type":"ContainerDied","Data":"f8ccb1f40bbad9cb079ba838cfe16de54befa657987e6cca0670a0e1e8842c12"}
Nov 01 04:30:05.190545 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.190531    4882 scope.go:122] "RemoveContainer" containerID="f8ccb1f40bbad9cb079ba838cfe16de54befa657987e6cca0670a0e1e8842c12"
Nov 01 04:30:05.190748 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:05.190733    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h_odh-ai-gateway-infra(5c1a2d6a-5672-4106-9332-59330be63eec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" podUID="5c1a2d6a-5672-4106-9332-59330be63eec"
Nov 01 04:30:05.202450 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.202418    4882 scope.go:122] "RemoveContainer" containerID="8571ae7a5c5ded72f4ed6397806dd62686b719b0fe20457145930254eef26bc3"
Nov 01 04:30:05.216280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.216262    4882 scope.go:122] "RemoveContainer" containerID="200dc5a275932be009823faf7e0d11464e5e79f9bbfe736e14a030b466ffb53a"
Nov 01 04:30:05.600044 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:05.600012    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9390f9be-7ca3-4bf4-9da5-65cdc7daba97" path="/var/lib/kubelet/pods/9390f9be-7ca3-4bf4-9da5-65cdc7daba97/volumes"
Nov 01 04:30:06.199914 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.199881    4882 scope.go:122] "RemoveContainer" containerID="a3508c48c2b6bfd7c5f0608c17a86b4d5220bfde221f63198c38819ff662a30b"
Nov 01 04:30:06.200379 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:06.200209    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx_odh-ai-gateway-infra(bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c"
Nov 01 04:30:06.201524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.201506    4882 scope.go:122] "RemoveContainer" containerID="f8ccb1f40bbad9cb079ba838cfe16de54befa657987e6cca0670a0e1e8842c12"
Nov 01 04:30:06.201782 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:06.201755    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h_odh-ai-gateway-infra(5c1a2d6a-5672-4106-9332-59330be63eec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" podUID="5c1a2d6a-5672-4106-9332-59330be63eec"
Nov 01 04:30:06.203524 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.203101    4882 scope.go:122] "RemoveContainer" containerID="230890e85439c1b00a321830d957e4e84fec59b776e5699e90bb765502293c43"
Nov 01 04:30:06.203524 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:06.203376    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr_odh-ai-gateway-infra(ba6a23a3-2d0f-4083-a42e-3067dbe62be7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7"
Nov 01 04:30:06.357741 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.357715    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:06.530476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.530376    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb84c12-e9cf-4428-b367-e617b91f4c03-kube-api-access-rck65\" (UniqueName: \"kubernetes.io/projected/8bb84c12-e9cf-4428-b367-e617b91f4c03-kube-api-access-rck65\") pod \"8bb84c12-e9cf-4428-b367-e617b91f4c03\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") "
Nov 01 04:30:06.530476 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.530413    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-xks-ca\") pod \"8bb84c12-e9cf-4428-b367-e617b91f4c03\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") "
Nov 01 04:30:06.530679 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.530518    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-openshift-ca\") pod \"8bb84c12-e9cf-4428-b367-e617b91f4c03\" (UID: \"8bb84c12-e9cf-4428-b367-e617b91f4c03\") "
Nov 01 04:30:06.530908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.530881    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-openshift-ca" pod "8bb84c12-e9cf-4428-b367-e617b91f4c03" (UID: "8bb84c12-e9cf-4428-b367-e617b91f4c03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:06.532535 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.532515    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-xks-ca" pod "8bb84c12-e9cf-4428-b367-e617b91f4c03" (UID: "8bb84c12-e9cf-4428-b367-e617b91f4c03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:30:06.532742 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.532717    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb84c12-e9cf-4428-b367-e617b91f4c03-kube-api-access-rck65" pod "8bb84c12-e9cf-4428-b367-e617b91f4c03" (UID: "8bb84c12-e9cf-4428-b367-e617b91f4c03"). InnerVolumeSpecName "kube-api-access-rck65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:06.632475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.632408    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:06.632475 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.632472    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rck65\" (UniqueName: \"kubernetes.io/projected/8bb84c12-e9cf-4428-b367-e617b91f4c03-kube-api-access-rck65\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:06.632688 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:06.632489    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb84c12-e9cf-4428-b367-e617b91f4c03-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:07.210246 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:07.210211    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"
Nov 01 04:30:07.210665 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:07.210244    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2" event={"ID":"8bb84c12-e9cf-4428-b367-e617b91f4c03","Type":"ContainerDied","Data":"faf4688e5bd2cdd25b4d9b2eb1ca0325854e719b8903c98a1d5dde34c2d0e057"}
Nov 01 04:30:07.210665 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:07.210296    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faf4688e5bd2cdd25b4d9b2eb1ca0325854e719b8903c98a1d5dde34c2d0e057"
Nov 01 04:30:09.593609 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.593545    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:30:09.767370 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.767339    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istiod-ca-cert\") pod \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") "
Nov 01 04:30:09.767573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.767406    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-data\") pod \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") "
Nov 01 04:30:09.767573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.767472    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-certs\") pod \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") "
Nov 01 04:30:09.767573 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.767539    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-token\" (UniqueName: \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-token\") pod \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") "
Nov 01 04:30:09.767706 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.767596    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-kube-api-access-x4wqk\" (UniqueName: \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-kube-api-access-x4wqk\") pod \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") "
Nov 01 04:30:09.767706 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.767645    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-credential-socket\") pod \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") "
Nov 01 04:30:09.767706 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.767697    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-envoy\") pod \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") "
Nov 01 04:30:09.767812 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.767725    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-data" pod "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" (UID: "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:09.767812 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.767765    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istiod-ca-cert" pod "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" (UID: "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:09.768057 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.768035    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-credential-socket" pod "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" (UID: "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:09.768238 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.768200    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-socket\") pod \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") "
Nov 01 04:30:09.768238 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.768245    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-podinfo\") pod \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\" (UID: \"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac\") "
Nov 01 04:30:09.768498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.768247    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-certs" pod "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" (UID: "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:09.768498 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.768405    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-socket" pod "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" (UID: "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:09.768872 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.768849    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:09.768983 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.768878    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:09.768983 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.768893    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:09.768983 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.768907    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:09.768983 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.768920    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:09.770253 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.770230    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-envoy" pod "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" (UID: "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:09.770657 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.770637    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-token" pod "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" (UID: "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:09.770751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.770635    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-kube-api-access-x4wqk" pod "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" (UID: "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac"). InnerVolumeSpecName "kube-api-access-x4wqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:09.775199 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.775176    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-podinfo" pod "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" (UID: "3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:30:09.819468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.819424    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"]
Nov 01 04:30:09.870520 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.870451    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:09.870520 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.870477    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:09.870520 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.870486    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:09.870520 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:09.870495    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4wqk\" (UniqueName: \"kubernetes.io/projected/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac-kube-api-access-x4wqk\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:10.231472 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.231439    4882 generic.go:356] "Generic (PLEG): container finished" podID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerID="f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe" exitCode=0
Nov 01 04:30:10.231632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.231503    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"
Nov 01 04:30:10.231632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.231522    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" event={"ID":"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac","Type":"ContainerDied","Data":"f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe"}
Nov 01 04:30:10.231632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.231584    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5" event={"ID":"3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac","Type":"ContainerDied","Data":"c1f3f08a4f6bc46ee13815b097293ad8f7612a9afa3d90101108a1d1e2454a8b"}
Nov 01 04:30:10.231632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.231608    4882 scope.go:122] "RemoveContainer" containerID="f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe"
Nov 01 04:30:10.242007 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.241995    4882 scope.go:122] "RemoveContainer" containerID="f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a"
Nov 01 04:30:10.255010 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.252389    4882 scope.go:122] "RemoveContainer" containerID="f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe"
Nov 01 04:30:10.255010 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:10.252723    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe\": container with ID starting with f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe not found: ID does not exist" containerID="f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe"
Nov 01 04:30:10.255010 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.252754    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe"} err="failed to get container status \"f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe\": rpc error: code = NotFound desc = could not find container \"f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe\": container with ID starting with f77c6559fc62c2e5e463656a9afa86ea53f69dde892e276274ecb84440242bbe not found: ID does not exist"
Nov 01 04:30:10.255010 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.252778    4882 scope.go:122] "RemoveContainer" containerID="f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a"
Nov 01 04:30:10.255010 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:10.253044    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a\": container with ID starting with f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a not found: ID does not exist" containerID="f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a"
Nov 01 04:30:10.255010 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.253072    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a"} err="failed to get container status \"f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a\": rpc error: code = NotFound desc = could not find container \"f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a\": container with ID starting with f35fe9951d112d3968c9eaf1cd8544ac7c381e26ce92615398f14de6ddf0d39a not found: ID does not exist"
Nov 01 04:30:10.386336 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.386302    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"]
Nov 01 04:30:10.392077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:10.392048    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-804014-openshift-default-76dc5db6b9-prkw5"]
Nov 01 04:30:11.601518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:11.601490    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" path="/var/lib/kubelet/pods/3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac/volumes"
Nov 01 04:30:13.770382 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:13.770352    4882 scope.go:122] "RemoveContainer" containerID="d58071b6faa933c059724ebf79be55b449e1d334154dfb83ac2b37e1a826b855"
Nov 01 04:30:13.780183 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:13.780163    4882 scope.go:122] "RemoveContainer" containerID="e3a743bb9288e666586cf196a10aefbc1b380f86db8bf9fd003baf379d1304c1"
Nov 01 04:30:15.593241 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:15.593207    4882 scope.go:122] "RemoveContainer" containerID="230890e85439c1b00a321830d957e4e84fec59b776e5699e90bb765502293c43"
Nov 01 04:30:15.593670 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:15.593279    4882 scope.go:122] "RemoveContainer" containerID="be763ab98747aa19180ee9707cc5bc3229be4187dd4f9836f3ae65ed1a5c685a"
Nov 01 04:30:15.593670 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:15.593376    4882 scope.go:122] "RemoveContainer" containerID="f8ccb1f40bbad9cb079ba838cfe16de54befa657987e6cca0670a0e1e8842c12"
Nov 01 04:30:16.277226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:16.277192    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" event={"ID":"5c1a2d6a-5672-4106-9332-59330be63eec","Type":"ContainerStarted","Data":"c979016c2269dfb6a8d853ec7362d3fe52c7e88e7eb5e032e5cab6609898e46c"}
Nov 01 04:30:16.279257 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:16.279226    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" event={"ID":"ba6a23a3-2d0f-4083-a42e-3067dbe62be7","Type":"ContainerStarted","Data":"bed86bc5b310b0dfbdbff652fd2f9e48aa849ccb16754ba06b7fb0d7a735b43b"}
Nov 01 04:30:16.281004 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:16.280982    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m" event={"ID":"53d9aefe-476b-49e5-8b67-7f8cb2768be8","Type":"ContainerStarted","Data":"faf0cfb9b1068e655f6d2dc716115ab11ffe4e3aade7be94bc42a2d5ff4f4ed8"}
Nov 01 04:30:16.593373 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:16.593339    4882 scope.go:122] "RemoveContainer" containerID="482dbf3061a2f1b2e4f08868e7d50f837094bdcdef16628d57a4e5dc184045df"
Nov 01 04:30:17.289631 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.289542    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2" event={"ID":"5b73bbcd-28bf-401f-97f4-301dd26d4c46","Type":"ContainerStarted","Data":"1ed7ae57f41d83b667061b07a2faa9a78068b2caca362857e81c0f4a6e90f172"}
Nov 01 04:30:17.291499 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.291473    4882 generic.go:356] "Generic (PLEG): container finished" podID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerID="faf0cfb9b1068e655f6d2dc716115ab11ffe4e3aade7be94bc42a2d5ff4f4ed8" exitCode=60
Nov 01 04:30:17.291610 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.291543    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m" event={"ID":"53d9aefe-476b-49e5-8b67-7f8cb2768be8","Type":"ContainerDied","Data":"faf0cfb9b1068e655f6d2dc716115ab11ffe4e3aade7be94bc42a2d5ff4f4ed8"}
Nov 01 04:30:17.291610 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.291595    4882 scope.go:122] "RemoveContainer" containerID="be763ab98747aa19180ee9707cc5bc3229be4187dd4f9836f3ae65ed1a5c685a"
Nov 01 04:30:17.291961 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.291944    4882 scope.go:122] "RemoveContainer" containerID="faf0cfb9b1068e655f6d2dc716115ab11ffe4e3aade7be94bc42a2d5ff4f4ed8"
Nov 01 04:30:17.292269 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:17.292238    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m_odh-ai-gateway-infra(53d9aefe-476b-49e5-8b67-7f8cb2768be8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8"
Nov 01 04:30:17.294238 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.294220    4882 generic.go:356] "Generic (PLEG): container finished" podID="5c1a2d6a-5672-4106-9332-59330be63eec" containerID="c979016c2269dfb6a8d853ec7362d3fe52c7e88e7eb5e032e5cab6609898e46c" exitCode=60
Nov 01 04:30:17.294335 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.294294    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" event={"ID":"5c1a2d6a-5672-4106-9332-59330be63eec","Type":"ContainerDied","Data":"c979016c2269dfb6a8d853ec7362d3fe52c7e88e7eb5e032e5cab6609898e46c"}
Nov 01 04:30:17.294680 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.294653    4882 scope.go:122] "RemoveContainer" containerID="c979016c2269dfb6a8d853ec7362d3fe52c7e88e7eb5e032e5cab6609898e46c"
Nov 01 04:30:17.294944 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:17.294886    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h_odh-ai-gateway-infra(5c1a2d6a-5672-4106-9332-59330be63eec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" podUID="5c1a2d6a-5672-4106-9332-59330be63eec"
Nov 01 04:30:17.296188 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.296169    4882 generic.go:356] "Generic (PLEG): container finished" podID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerID="bed86bc5b310b0dfbdbff652fd2f9e48aa849ccb16754ba06b7fb0d7a735b43b" exitCode=60
Nov 01 04:30:17.296297 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.296247    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" event={"ID":"ba6a23a3-2d0f-4083-a42e-3067dbe62be7","Type":"ContainerDied","Data":"bed86bc5b310b0dfbdbff652fd2f9e48aa849ccb16754ba06b7fb0d7a735b43b"}
Nov 01 04:30:17.296540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.296525    4882 scope.go:122] "RemoveContainer" containerID="bed86bc5b310b0dfbdbff652fd2f9e48aa849ccb16754ba06b7fb0d7a735b43b"
Nov 01 04:30:17.296735 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:17.296717    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr_odh-ai-gateway-infra(ba6a23a3-2d0f-4083-a42e-3067dbe62be7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7"
Nov 01 04:30:17.304366 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.304347    4882 scope.go:122] "RemoveContainer" containerID="f8ccb1f40bbad9cb079ba838cfe16de54befa657987e6cca0670a0e1e8842c12"
Nov 01 04:30:17.317227 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.317205    4882 scope.go:122] "RemoveContainer" containerID="230890e85439c1b00a321830d957e4e84fec59b776e5699e90bb765502293c43"
Nov 01 04:30:17.395518 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.395484    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"]
Nov 01 04:30:17.399587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.399537    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2" podStartSLOduration=17.399524271 podStartE2EDuration="17.399524271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:30:17.396192269 +0000 UTC m=+2885.958225286" watchObservedRunningTime="2026-11-01 04:30:17.399524271 +0000 UTC m=+2885.961557287"
Nov 01 04:30:17.505019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.504986    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"]
Nov 01 04:30:17.592719 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.592687    4882 scope.go:122] "RemoveContainer" containerID="aba31400f6c0343b3fe91b68f43a68beea4bf9c80c3606b9103b23e0d34be750"
Nov 01 04:30:17.613565 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:17.613536    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"]
Nov 01 04:30:18.307680 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.307649    4882 generic.go:356] "Generic (PLEG): container finished" podID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerID="1ed7ae57f41d83b667061b07a2faa9a78068b2caca362857e81c0f4a6e90f172" exitCode=60
Nov 01 04:30:18.307790 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.307716    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2" event={"ID":"5b73bbcd-28bf-401f-97f4-301dd26d4c46","Type":"ContainerDied","Data":"1ed7ae57f41d83b667061b07a2faa9a78068b2caca362857e81c0f4a6e90f172"}
Nov 01 04:30:18.307855 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.307791    4882 scope.go:122] "RemoveContainer" containerID="482dbf3061a2f1b2e4f08868e7d50f837094bdcdef16628d57a4e5dc184045df"
Nov 01 04:30:18.308228 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.308204    4882 scope.go:122] "RemoveContainer" containerID="1ed7ae57f41d83b667061b07a2faa9a78068b2caca362857e81c0f4a6e90f172"
Nov 01 04:30:18.308562 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:18.308542    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891790-7z4b2_odh-ai-gateway-infra(5b73bbcd-28bf-401f-97f4-301dd26d4c46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46"
Nov 01 04:30:18.309835 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.309816    4882 generic.go:356] "Generic (PLEG): container finished" podID="e5771967-299c-4e10-bd92-2637c3ec175b" containerID="66ef95563ed990fcc19f136e1682c74776a0b1bf637dfe1cf6eeaa0b5b6b370c" exitCode=60
Nov 01 04:30:18.309936 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.309892    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w" event={"ID":"e5771967-299c-4e10-bd92-2637c3ec175b","Type":"ContainerDied","Data":"66ef95563ed990fcc19f136e1682c74776a0b1bf637dfe1cf6eeaa0b5b6b370c"}
Nov 01 04:30:18.310272 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.310254    4882 scope.go:122] "RemoveContainer" containerID="66ef95563ed990fcc19f136e1682c74776a0b1bf637dfe1cf6eeaa0b5b6b370c"
Nov 01 04:30:18.310588 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:18.310564    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w_odh-ai-gateway-infra(e5771967-299c-4e10-bd92-2637c3ec175b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w" podUID="e5771967-299c-4e10-bd92-2637c3ec175b"
Nov 01 04:30:18.339211 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.339170    4882 scope.go:122] "RemoveContainer" containerID="aba31400f6c0343b3fe91b68f43a68beea4bf9c80c3606b9103b23e0d34be750"
Nov 01 04:30:18.408739 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.408690    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"]
Nov 01 04:30:18.484583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.484551    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"]
Nov 01 04:30:18.527550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.527486    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"
Nov 01 04:30:18.531096 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.531075    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"
Nov 01 04:30:18.534751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.534725    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"
Nov 01 04:30:18.574680 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.574643    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1a2d6a-5672-4106-9332-59330be63eec-kube-api-access-zpb4t\" (UniqueName: \"kubernetes.io/projected/5c1a2d6a-5672-4106-9332-59330be63eec-kube-api-access-zpb4t\") pod \"5c1a2d6a-5672-4106-9332-59330be63eec\" (UID: \"5c1a2d6a-5672-4106-9332-59330be63eec\") "
Nov 01 04:30:18.574868 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.574779    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-kube-api-access-p4rgt\" (UniqueName: \"kubernetes.io/projected/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-kube-api-access-p4rgt\") pod \"ba6a23a3-2d0f-4083-a42e-3067dbe62be7\" (UID: \"ba6a23a3-2d0f-4083-a42e-3067dbe62be7\") "
Nov 01 04:30:18.574917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.574871    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-maas-api-service-ca\") pod \"ba6a23a3-2d0f-4083-a42e-3067dbe62be7\" (UID: \"ba6a23a3-2d0f-4083-a42e-3067dbe62be7\") "
Nov 01 04:30:18.574917 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.574907    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53d9aefe-476b-49e5-8b67-7f8cb2768be8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d9aefe-476b-49e5-8b67-7f8cb2768be8-maas-api-service-ca\") pod \"53d9aefe-476b-49e5-8b67-7f8cb2768be8\" (UID: \"53d9aefe-476b-49e5-8b67-7f8cb2768be8\") "
Nov 01 04:30:18.575002 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.574960    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1a2d6a-5672-4106-9332-59330be63eec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1a2d6a-5672-4106-9332-59330be63eec-maas-api-service-ca\") pod \"5c1a2d6a-5672-4106-9332-59330be63eec\" (UID: \"5c1a2d6a-5672-4106-9332-59330be63eec\") "
Nov 01 04:30:18.575002 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.574996    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d9aefe-476b-49e5-8b67-7f8cb2768be8-kube-api-access-vnfpm\" (UniqueName: \"kubernetes.io/projected/53d9aefe-476b-49e5-8b67-7f8cb2768be8-kube-api-access-vnfpm\") pod \"53d9aefe-476b-49e5-8b67-7f8cb2768be8\" (UID: \"53d9aefe-476b-49e5-8b67-7f8cb2768be8\") "
Nov 01 04:30:18.575278 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.575251    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-maas-api-service-ca" pod "ba6a23a3-2d0f-4083-a42e-3067dbe62be7" (UID: "ba6a23a3-2d0f-4083-a42e-3067dbe62be7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:18.575395 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.575341    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1a2d6a-5672-4106-9332-59330be63eec-maas-api-service-ca" pod "5c1a2d6a-5672-4106-9332-59330be63eec" (UID: "5c1a2d6a-5672-4106-9332-59330be63eec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:18.575606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.575582    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:18.575716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.575609    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c1a2d6a-5672-4106-9332-59330be63eec-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:18.575716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.575703    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53d9aefe-476b-49e5-8b67-7f8cb2768be8-maas-api-service-ca" pod "53d9aefe-476b-49e5-8b67-7f8cb2768be8" (UID: "53d9aefe-476b-49e5-8b67-7f8cb2768be8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:18.577088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.577066    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1a2d6a-5672-4106-9332-59330be63eec-kube-api-access-zpb4t" pod "5c1a2d6a-5672-4106-9332-59330be63eec" (UID: "5c1a2d6a-5672-4106-9332-59330be63eec"). InnerVolumeSpecName "kube-api-access-zpb4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:18.577170 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.577155    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d9aefe-476b-49e5-8b67-7f8cb2768be8-kube-api-access-vnfpm" pod "53d9aefe-476b-49e5-8b67-7f8cb2768be8" (UID: "53d9aefe-476b-49e5-8b67-7f8cb2768be8"). InnerVolumeSpecName "kube-api-access-vnfpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:18.577393 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.577378    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-kube-api-access-p4rgt" pod "ba6a23a3-2d0f-4083-a42e-3067dbe62be7" (UID: "ba6a23a3-2d0f-4083-a42e-3067dbe62be7"). InnerVolumeSpecName "kube-api-access-p4rgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:18.586844 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.586815    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"]
Nov 01 04:30:18.592588 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.592520    4882 scope.go:122] "RemoveContainer" containerID="a3508c48c2b6bfd7c5f0608c17a86b4d5220bfde221f63198c38819ff662a30b"
Nov 01 04:30:18.676513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.676483    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53d9aefe-476b-49e5-8b67-7f8cb2768be8-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:18.676513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.676509    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnfpm\" (UniqueName: \"kubernetes.io/projected/53d9aefe-476b-49e5-8b67-7f8cb2768be8-kube-api-access-vnfpm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:18.676513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.676520    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpb4t\" (UniqueName: \"kubernetes.io/projected/5c1a2d6a-5672-4106-9332-59330be63eec-kube-api-access-zpb4t\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:18.677017 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:18.676534    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4rgt\" (UniqueName: \"kubernetes.io/projected/ba6a23a3-2d0f-4083-a42e-3067dbe62be7-kube-api-access-p4rgt\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:19.325627 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.325592    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"
Nov 01 04:30:19.325835 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.325599    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr" event={"ID":"ba6a23a3-2d0f-4083-a42e-3067dbe62be7","Type":"ContainerDied","Data":"18e66932345266af0f23b1ba0f07b0f350ca59bbec26923236baae240cbd2bc6"}
Nov 01 04:30:19.325835 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.325782    4882 scope.go:122] "RemoveContainer" containerID="bed86bc5b310b0dfbdbff652fd2f9e48aa849ccb16754ba06b7fb0d7a735b43b"
Nov 01 04:30:19.334956 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.334927    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" event={"ID":"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c","Type":"ContainerStarted","Data":"223e193aecfad9d3768d6944e9e0290070f33d934de0b4e40f3c9b5f7fd5ebe7"}
Nov 01 04:30:19.343030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.336468    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m" event={"ID":"53d9aefe-476b-49e5-8b67-7f8cb2768be8","Type":"ContainerDied","Data":"984150c90d82b32fc8da5e3d60b7d1c2e472d5bb8f4e3c8db71a53d4b7131b91"}
Nov 01 04:30:19.343030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.336495    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"
Nov 01 04:30:19.343030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.337158    4882 scope.go:122] "RemoveContainer" containerID="faf0cfb9b1068e655f6d2dc716115ab11ffe4e3aade7be94bc42a2d5ff4f4ed8"
Nov 01 04:30:19.343030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.338265    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h" event={"ID":"5c1a2d6a-5672-4106-9332-59330be63eec","Type":"ContainerDied","Data":"11b62e356be1ca5b63b22f9ff352127c8bb682d5d1c320498f8a1a8dceba0951"}
Nov 01 04:30:19.343030 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.338340    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"
Nov 01 04:30:19.347330 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.347234    4882 scope.go:122] "RemoveContainer" containerID="c979016c2269dfb6a8d853ec7362d3fe52c7e88e7eb5e032e5cab6609898e46c"
Nov 01 04:30:19.488008 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.487983    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"]
Nov 01 04:30:19.494226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.494203    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fb1c1f-29891790-xkj7h"]
Nov 01 04:30:19.542819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.542791    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"
Nov 01 04:30:19.592026 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.591900    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b73bbcd-28bf-401f-97f4-301dd26d4c46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b73bbcd-28bf-401f-97f4-301dd26d4c46-maas-api-service-ca\") pod \"5b73bbcd-28bf-401f-97f4-301dd26d4c46\" (UID: \"5b73bbcd-28bf-401f-97f4-301dd26d4c46\") "
Nov 01 04:30:19.592217 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.592046    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b73bbcd-28bf-401f-97f4-301dd26d4c46-kube-api-access-cw7kk\" (UniqueName: \"kubernetes.io/projected/5b73bbcd-28bf-401f-97f4-301dd26d4c46-kube-api-access-cw7kk\") pod \"5b73bbcd-28bf-401f-97f4-301dd26d4c46\" (UID: \"5b73bbcd-28bf-401f-97f4-301dd26d4c46\") "
Nov 01 04:30:19.592775 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.592745    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b73bbcd-28bf-401f-97f4-301dd26d4c46-maas-api-service-ca" pod "5b73bbcd-28bf-401f-97f4-301dd26d4c46" (UID: "5b73bbcd-28bf-401f-97f4-301dd26d4c46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:19.595150 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.595106    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b73bbcd-28bf-401f-97f4-301dd26d4c46-kube-api-access-cw7kk" pod "5b73bbcd-28bf-401f-97f4-301dd26d4c46" (UID: "5b73bbcd-28bf-401f-97f4-301dd26d4c46"). InnerVolumeSpecName "kube-api-access-cw7kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:19.607437 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.607393    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1a2d6a-5672-4106-9332-59330be63eec" path="/var/lib/kubelet/pods/5c1a2d6a-5672-4106-9332-59330be63eec/volumes"
Nov 01 04:30:19.608148 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.608129    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"]
Nov 01 04:30:19.608233 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.608171    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2c259-29891790-csggr"]
Nov 01 04:30:19.639645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.639608    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"]
Nov 01 04:30:19.693317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.693279    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw7kk\" (UniqueName: \"kubernetes.io/projected/5b73bbcd-28bf-401f-97f4-301dd26d4c46-kube-api-access-cw7kk\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:19.693317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.693313    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b73bbcd-28bf-401f-97f4-301dd26d4c46-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:19.790809 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.790781    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"
Nov 01 04:30:19.809959 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.809928    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"]
Nov 01 04:30:19.813276 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.813247    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a9c5b3-29891790-ddv6m"]
Nov 01 04:30:19.895080 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.894995    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5771967-299c-4e10-bd92-2637c3ec175b-kube-api-access-9c4f2\" (UniqueName: \"kubernetes.io/projected/e5771967-299c-4e10-bd92-2637c3ec175b-kube-api-access-9c4f2\") pod \"e5771967-299c-4e10-bd92-2637c3ec175b\" (UID: \"e5771967-299c-4e10-bd92-2637c3ec175b\") "
Nov 01 04:30:19.895241 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.895178    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5771967-299c-4e10-bd92-2637c3ec175b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5771967-299c-4e10-bd92-2637c3ec175b-maas-api-service-ca\") pod \"e5771967-299c-4e10-bd92-2637c3ec175b\" (UID: \"e5771967-299c-4e10-bd92-2637c3ec175b\") "
Nov 01 04:30:19.895569 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.895538    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5771967-299c-4e10-bd92-2637c3ec175b-maas-api-service-ca" pod "e5771967-299c-4e10-bd92-2637c3ec175b" (UID: "e5771967-299c-4e10-bd92-2637c3ec175b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:19.897233 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.897201    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5771967-299c-4e10-bd92-2637c3ec175b-kube-api-access-9c4f2" pod "e5771967-299c-4e10-bd92-2637c3ec175b" (UID: "e5771967-299c-4e10-bd92-2637c3ec175b"). InnerVolumeSpecName "kube-api-access-9c4f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:19.996239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.996201    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c4f2\" (UniqueName: \"kubernetes.io/projected/e5771967-299c-4e10-bd92-2637c3ec175b-kube-api-access-9c4f2\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:19.996239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:19.996239    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5771967-299c-4e10-bd92-2637c3ec175b-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:20.346308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.346270    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"
Nov 01 04:30:20.346509 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.346271    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2" event={"ID":"5b73bbcd-28bf-401f-97f4-301dd26d4c46","Type":"ContainerDied","Data":"f73778a19417dbfa95f5d3f0d2b907eb0538e5b07c91abd1aed86dc2eb2dd1aa"}
Nov 01 04:30:20.346509 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.346458    4882 scope.go:122] "RemoveContainer" containerID="1ed7ae57f41d83b667061b07a2faa9a78068b2caca362857e81c0f4a6e90f172"
Nov 01 04:30:20.347874 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.347852    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"
Nov 01 04:30:20.347978 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.347872    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w" event={"ID":"e5771967-299c-4e10-bd92-2637c3ec175b","Type":"ContainerDied","Data":"5db270c0cbe672bf0a0cf7caa33536446f708c359fefceb2bc097e2455a78314"}
Nov 01 04:30:20.350301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.350279    4882 generic.go:356] "Generic (PLEG): container finished" podID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerID="223e193aecfad9d3768d6944e9e0290070f33d934de0b4e40f3c9b5f7fd5ebe7" exitCode=60
Nov 01 04:30:20.350411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.350356    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" event={"ID":"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c","Type":"ContainerDied","Data":"223e193aecfad9d3768d6944e9e0290070f33d934de0b4e40f3c9b5f7fd5ebe7"}
Nov 01 04:30:20.350751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.350734    4882 scope.go:122] "RemoveContainer" containerID="223e193aecfad9d3768d6944e9e0290070f33d934de0b4e40f3c9b5f7fd5ebe7"
Nov 01 04:30:20.351017 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:20.350998    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx_odh-ai-gateway-infra(bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c"
Nov 01 04:30:20.357622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.357602    4882 scope.go:122] "RemoveContainer" containerID="66ef95563ed990fcc19f136e1682c74776a0b1bf637dfe1cf6eeaa0b5b6b370c"
Nov 01 04:30:20.366228 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.366208    4882 scope.go:122] "RemoveContainer" containerID="a3508c48c2b6bfd7c5f0608c17a86b4d5220bfde221f63198c38819ff662a30b"
Nov 01 04:30:20.499387 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.499353    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"]
Nov 01 04:30:20.503223 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.503190    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891790-7z4b2"]
Nov 01 04:30:20.603823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.603733    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"]
Nov 01 04:30:20.607371 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.607331    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f7545f-29891790-jkt6w"]
Nov 01 04:30:20.720347 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:20.720311    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"]
Nov 01 04:30:21.529897 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:21.529866    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"
Nov 01 04:30:21.599341 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:21.599303    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" path="/var/lib/kubelet/pods/53d9aefe-476b-49e5-8b67-7f8cb2768be8/volumes"
Nov 01 04:30:21.599786 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:21.599770    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" path="/var/lib/kubelet/pods/5b73bbcd-28bf-401f-97f4-301dd26d4c46/volumes"
Nov 01 04:30:21.600144 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:21.600131    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" path="/var/lib/kubelet/pods/ba6a23a3-2d0f-4083-a42e-3067dbe62be7/volumes"
Nov 01 04:30:21.600809 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:21.600794    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5771967-299c-4e10-bd92-2637c3ec175b" path="/var/lib/kubelet/pods/e5771967-299c-4e10-bd92-2637c3ec175b/volumes"
Nov 01 04:30:21.616013 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:21.615985    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-maas-api-service-ca\") pod \"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c\" (UID: \"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c\") "
Nov 01 04:30:21.616189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:21.616174    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-kube-api-access-dtcmg\" (UniqueName: \"kubernetes.io/projected/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-kube-api-access-dtcmg\") pod \"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c\" (UID: \"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c\") "
Nov 01 04:30:21.616301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:21.616279    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-maas-api-service-ca" pod "bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" (UID: "bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:21.616511 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:21.616495    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-maas-api-service-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:21.618200 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:21.618180    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-kube-api-access-dtcmg" pod "bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" (UID: "bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c"). InnerVolumeSpecName "kube-api-access-dtcmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:21.717121 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:21.717090    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtcmg\" (UniqueName: \"kubernetes.io/projected/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c-kube-api-access-dtcmg\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:22.384900 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:22.384869    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"
Nov 01 04:30:22.384900 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:22.384894    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx" event={"ID":"bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c","Type":"ContainerDied","Data":"1f5a61e422297cbaf8d9b6477fae8499b6b6c0adecaf8490d55fb657a65a148b"}
Nov 01 04:30:22.385423 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:22.384962    4882 scope.go:122] "RemoveContainer" containerID="223e193aecfad9d3768d6944e9e0290070f33d934de0b4e40f3c9b5f7fd5ebe7"
Nov 01 04:30:22.538821 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:22.538782    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"]
Nov 01 04:30:22.584660 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:22.584626    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891790-cb7jx"]
Nov 01 04:30:23.485147 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.485092    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x" podUID="a07e6f0c-683d-44ae-9e25-db523c29b6de" containerName="payload-processing" containerID="cri-o://9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13" gracePeriod=25
Nov 01 04:30:23.587517 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.587473    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v" podUID="e5e9aad9-b32e-438e-a2ec-748a6443f58e" containerName="payload-pre-processing" containerID="cri-o://0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862" gracePeriod=25
Nov 01 04:30:23.600163 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.600134    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" path="/var/lib/kubelet/pods/bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c/volumes"
Nov 01 04:30:23.748868 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.748806    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:30:23.839385 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.839349    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07e6f0c-683d-44ae-9e25-db523c29b6de-kube-api-access-d7kr6\" (UniqueName: \"kubernetes.io/projected/a07e6f0c-683d-44ae-9e25-db523c29b6de-kube-api-access-d7kr6\") pod \"a07e6f0c-683d-44ae-9e25-db523c29b6de\" (UID: \"a07e6f0c-683d-44ae-9e25-db523c29b6de\") "
Nov 01 04:30:23.839592 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.839471    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07e6f0c-683d-44ae-9e25-db523c29b6de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07e6f0c-683d-44ae-9e25-db523c29b6de-plugins-config-volume\") pod \"a07e6f0c-683d-44ae-9e25-db523c29b6de\" (UID: \"a07e6f0c-683d-44ae-9e25-db523c29b6de\") "
Nov 01 04:30:23.839903 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.839875    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07e6f0c-683d-44ae-9e25-db523c29b6de-plugins-config-volume" pod "a07e6f0c-683d-44ae-9e25-db523c29b6de" (UID: "a07e6f0c-683d-44ae-9e25-db523c29b6de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:23.841645 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.841621    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07e6f0c-683d-44ae-9e25-db523c29b6de-kube-api-access-d7kr6" pod "a07e6f0c-683d-44ae-9e25-db523c29b6de" (UID: "a07e6f0c-683d-44ae-9e25-db523c29b6de"). InnerVolumeSpecName "kube-api-access-d7kr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:23.861823 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.861793    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:30:23.941118 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.941089    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e9aad9-b32e-438e-a2ec-748a6443f58e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e9aad9-b32e-438e-a2ec-748a6443f58e-plugins-config-volume\") pod \"e5e9aad9-b32e-438e-a2ec-748a6443f58e\" (UID: \"e5e9aad9-b32e-438e-a2ec-748a6443f58e\") "
Nov 01 04:30:23.941265 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.941137    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e9aad9-b32e-438e-a2ec-748a6443f58e-kube-api-access-mhzs2\" (UniqueName: \"kubernetes.io/projected/e5e9aad9-b32e-438e-a2ec-748a6443f58e-kube-api-access-mhzs2\") pod \"e5e9aad9-b32e-438e-a2ec-748a6443f58e\" (UID: \"e5e9aad9-b32e-438e-a2ec-748a6443f58e\") "
Nov 01 04:30:23.941463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.941423    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07e6f0c-683d-44ae-9e25-db523c29b6de-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:23.941503 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.941456    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e9aad9-b32e-438e-a2ec-748a6443f58e-plugins-config-volume" pod "e5e9aad9-b32e-438e-a2ec-748a6443f58e" (UID: "e5e9aad9-b32e-438e-a2ec-748a6443f58e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:23.941503 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.941467    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7kr6\" (UniqueName: \"kubernetes.io/projected/a07e6f0c-683d-44ae-9e25-db523c29b6de-kube-api-access-d7kr6\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:23.943282 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:23.943257    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e9aad9-b32e-438e-a2ec-748a6443f58e-kube-api-access-mhzs2" pod "e5e9aad9-b32e-438e-a2ec-748a6443f58e" (UID: "e5e9aad9-b32e-438e-a2ec-748a6443f58e"). InnerVolumeSpecName "kube-api-access-mhzs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:24.042728 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.042649    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5e9aad9-b32e-438e-a2ec-748a6443f58e-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:24.042728 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.042676    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzs2\" (UniqueName: \"kubernetes.io/projected/e5e9aad9-b32e-438e-a2ec-748a6443f58e-kube-api-access-mhzs2\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:24.400571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.400486    4882 generic.go:356] "Generic (PLEG): container finished" podID="a07e6f0c-683d-44ae-9e25-db523c29b6de" containerID="9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13" exitCode=0
Nov 01 04:30:24.400571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.400533    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"
Nov 01 04:30:24.400804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.400575    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x" event={"ID":"a07e6f0c-683d-44ae-9e25-db523c29b6de","Type":"ContainerDied","Data":"9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13"}
Nov 01 04:30:24.400804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.400630    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x" event={"ID":"a07e6f0c-683d-44ae-9e25-db523c29b6de","Type":"ContainerDied","Data":"9917f2dae8dc3671d4dfd1e7e4cf67bede57c7f493a0eb39f6698ab8bbf52935"}
Nov 01 04:30:24.400804 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.400662    4882 scope.go:122] "RemoveContainer" containerID="9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13"
Nov 01 04:30:24.402112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.402085    4882 generic.go:356] "Generic (PLEG): container finished" podID="e5e9aad9-b32e-438e-a2ec-748a6443f58e" containerID="0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862" exitCode=0
Nov 01 04:30:24.402247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.402125    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"
Nov 01 04:30:24.402247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.402169    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v" event={"ID":"e5e9aad9-b32e-438e-a2ec-748a6443f58e","Type":"ContainerDied","Data":"0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862"}
Nov 01 04:30:24.402247 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.402215    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v" event={"ID":"e5e9aad9-b32e-438e-a2ec-748a6443f58e","Type":"ContainerDied","Data":"dc6f832ae8d829e802219483c963548f765c974eb67bb5c55a3051b2ed53ef62"}
Nov 01 04:30:24.410782 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.410764    4882 scope.go:122] "RemoveContainer" containerID="9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13"
Nov 01 04:30:24.411087 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:24.411065    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13\": container with ID starting with 9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13 not found: ID does not exist" containerID="9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13"
Nov 01 04:30:24.411182 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.411093    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13"} err="failed to get container status \"9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13\": rpc error: code = NotFound desc = could not find container \"9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13\": container with ID starting with 9c561ea0c02dd3c0136f0fd79798f90ec10472e0791a20f417b4ee05dcdc4d13 not found: ID does not exist"
Nov 01 04:30:24.411182 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.411109    4882 scope.go:122] "RemoveContainer" containerID="0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862"
Nov 01 04:30:24.419701 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.419683    4882 scope.go:122] "RemoveContainer" containerID="0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862"
Nov 01 04:30:24.419941 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:24.419923    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862\": container with ID starting with 0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862 not found: ID does not exist" containerID="0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862"
Nov 01 04:30:24.419991 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.419952    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862"} err="failed to get container status \"0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862\": rpc error: code = NotFound desc = could not find container \"0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862\": container with ID starting with 0db8d985196bbdb2ee841bbd6d0c4b785c2e3ecc95afc6bbb566308886da5862 not found: ID does not exist"
Nov 01 04:30:24.552732 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.552702    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"]
Nov 01 04:30:24.598674 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.598645    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b8319b-7c8cd46599-ssm8x"]
Nov 01 04:30:24.656907 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.656833    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"]
Nov 01 04:30:24.661294 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.661267    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b8319b-76d48b85cf-gst8v"]
Nov 01 04:30:24.840015 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.839975    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main" containerID="cri-o://5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30" gracePeriod=45
Nov 01 04:30:24.971478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:24.971442    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused"
Nov 01 04:30:25.102347 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.102322    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 04:30:25.155061 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155029    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-model-cache\") pod \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") "
Nov 01 04:30:25.155221 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155088    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-dshm\") pod \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") "
Nov 01 04:30:25.155221 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155148    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-home\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-home\") pod \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") "
Nov 01 04:30:25.155221 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155186    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tmp-dir\") pod \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") "
Nov 01 04:30:25.155398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155262    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca3d592c-9b95-465e-ae21-f5f9417eb20f-kube-api-access-fkq6p\" (UniqueName: \"kubernetes.io/projected/ca3d592c-9b95-465e-ae21-f5f9417eb20f-kube-api-access-fkq6p\") pod \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") "
Nov 01 04:30:25.155398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155330    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-model-cache" pod "ca3d592c-9b95-465e-ae21-f5f9417eb20f" (UID: "ca3d592c-9b95-465e-ae21-f5f9417eb20f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:25.155398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155360    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tls-certs\") pod \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\" (UID: \"ca3d592c-9b95-465e-ae21-f5f9417eb20f\") "
Nov 01 04:30:25.155584 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155509    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-home" pod "ca3d592c-9b95-465e-ae21-f5f9417eb20f" (UID: "ca3d592c-9b95-465e-ae21-f5f9417eb20f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:25.155584 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155523    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tmp-dir" pod "ca3d592c-9b95-465e-ae21-f5f9417eb20f" (UID: "ca3d592c-9b95-465e-ae21-f5f9417eb20f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:25.155791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155766    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-model-cache\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:25.155791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155790    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-home\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:25.156042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.155799    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tmp-dir\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:25.157586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.157562    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-dshm" pod "ca3d592c-9b95-465e-ae21-f5f9417eb20f" (UID: "ca3d592c-9b95-465e-ae21-f5f9417eb20f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:25.157702 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.157598    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3d592c-9b95-465e-ae21-f5f9417eb20f-kube-api-access-fkq6p" pod "ca3d592c-9b95-465e-ae21-f5f9417eb20f" (UID: "ca3d592c-9b95-465e-ae21-f5f9417eb20f"). InnerVolumeSpecName "kube-api-access-fkq6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:25.157702 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.157657    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tls-certs" pod "ca3d592c-9b95-465e-ae21-f5f9417eb20f" (UID: "ca3d592c-9b95-465e-ae21-f5f9417eb20f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:30:25.257630 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.257538    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3d592c-9b95-465e-ae21-f5f9417eb20f-tls-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:25.257630 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.257570    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3d592c-9b95-465e-ae21-f5f9417eb20f-dshm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:25.257630 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.257585    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkq6p\" (UniqueName: \"kubernetes.io/projected/ca3d592c-9b95-465e-ae21-f5f9417eb20f-kube-api-access-fkq6p\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:25.412031 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.411989    4882 generic.go:356] "Generic (PLEG): container finished" podID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerID="5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30" exitCode=0
Nov 01 04:30:25.412031 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.412032    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"
Nov 01 04:30:25.412264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.412061    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" event={"ID":"ca3d592c-9b95-465e-ae21-f5f9417eb20f","Type":"ContainerDied","Data":"5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30"}
Nov 01 04:30:25.412264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.412121    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8" event={"ID":"ca3d592c-9b95-465e-ae21-f5f9417eb20f","Type":"ContainerDied","Data":"9a9f625a73ecb922e94d4ad3791315b2b96b2297dffd24983536f36bd58bd201"}
Nov 01 04:30:25.412264 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.412145    4882 scope.go:122] "RemoveContainer" containerID="5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30"
Nov 01 04:30:25.424597 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.424580    4882 scope.go:122] "RemoveContainer" containerID="5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30"
Nov 01 04:30:25.424819 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:25.424802    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30\": container with ID starting with 5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30 not found: ID does not exist" containerID="5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30"
Nov 01 04:30:25.424914 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.424826    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30"} err="failed to get container status \"5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30\": rpc error: code = NotFound desc = could not find container \"5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30\": container with ID starting with 5857d33fd123437fe2a740cfeb8145194bb04cd12dffa1c8ef6424f1dae32f30 not found: ID does not exist"
Nov 01 04:30:25.564542 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.564460    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"]
Nov 01 04:30:25.570040 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.570013    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c2c259/simulator-w1-c2c259-kserve-578cf77794-26dc8"]
Nov 01 04:30:25.600366 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.600320    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07e6f0c-683d-44ae-9e25-db523c29b6de" path="/var/lib/kubelet/pods/a07e6f0c-683d-44ae-9e25-db523c29b6de/volumes"
Nov 01 04:30:25.601111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.601087    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" path="/var/lib/kubelet/pods/ca3d592c-9b95-465e-ae21-f5f9417eb20f/volumes"
Nov 01 04:30:25.601746 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:25.601721    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e9aad9-b32e-438e-a2ec-748a6443f58e" path="/var/lib/kubelet/pods/e5e9aad9-b32e-438e-a2ec-748a6443f58e/volumes"
Nov 01 04:30:27.325836 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.325792    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"]
Nov 01 04:30:27.326278 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326264    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy"
Nov 01 04:30:27.326320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326278    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy"
Nov 01 04:30:27.326320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326288    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerName="cleanup"
Nov 01 04:30:27.326320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326293    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerName="cleanup"
Nov 01 04:30:27.326320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326302    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e5771967-299c-4e10-bd92-2637c3ec175b" containerName="cleanup"
Nov 01 04:30:27.326320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326308    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5771967-299c-4e10-bd92-2637c3ec175b" containerName="cleanup"
Nov 01 04:30:27.326320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326317    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5c1a2d6a-5672-4106-9332-59330be63eec" containerName="cleanup"
Nov 01 04:30:27.326320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326322    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1a2d6a-5672-4106-9332-59330be63eec" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326329    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a07e6f0c-683d-44ae-9e25-db523c29b6de" containerName="payload-processing"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326334    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07e6f0c-683d-44ae-9e25-db523c29b6de" containerName="payload-processing"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326345    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e5e9aad9-b32e-438e-a2ec-748a6443f58e" containerName="payload-pre-processing"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326349    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e9aad9-b32e-438e-a2ec-748a6443f58e" containerName="payload-pre-processing"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326356    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326360    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326366    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326371    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326383    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5c1a2d6a-5672-4106-9332-59330be63eec" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326387    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1a2d6a-5672-4106-9332-59330be63eec" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326399    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326404    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326415    4882 cpu_manager.go:414] "containerMap: removing container" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326419    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326439    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326444    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326449    4882 cpu_manager.go:414] "containerMap: removing container" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326453    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326459    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326464    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326476    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326480    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326486    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8bb84c12-e9cf-4428-b367-e617b91f4c03" containerName="revoke-keys"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326490    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb84c12-e9cf-4428-b367-e617b91f4c03" containerName="revoke-keys"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326497    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e5771967-299c-4e10-bd92-2637c3ec175b" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326501    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5771967-299c-4e10-bd92-2637c3ec175b" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326513    4882 cpu_manager.go:414] "containerMap: removing container" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326517    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326525    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326529    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerName="cleanup"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326535    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main"
Nov 01 04:30:27.326553 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326539    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326606    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326614    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1a2d6a-5672-4106-9332-59330be63eec" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326621    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5771967-299c-4e10-bd92-2637c3ec175b" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326626    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326633    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326637    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326642    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326648    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326654    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326660    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326664    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb84c12-e9cf-4428-b367-e617b91f4c03" containerName="revoke-keys"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326669    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5771967-299c-4e10-bd92-2637c3ec175b" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326675    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d9aefe-476b-49e5-8b67-7f8cb2768be8" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326681    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca3d592c-9b95-465e-ae21-f5f9417eb20f" containerName="main"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326686    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4e2fcf-c53e-4b42-ad8c-e9f195fd0dac" containerName="istio-proxy"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326690    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326695    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5771967-299c-4e10-bd92-2637c3ec175b" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326699    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1a2d6a-5672-4106-9332-59330be63eec" containerName="cleanup"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326704    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07e6f0c-683d-44ae-9e25-db523c29b6de" containerName="payload-processing"
Nov 01 04:30:27.327487 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.326709    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e9aad9-b32e-438e-a2ec-748a6443f58e" containerName="payload-pre-processing"
Nov 01 04:30:27.331542 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.331524    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:27.373061 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373037    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"]
Nov 01 04:30:27.373531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373517    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerName="cleanup"
Nov 01 04:30:27.373593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373533    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerName="cleanup"
Nov 01 04:30:27.373593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373545    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e5771967-299c-4e10-bd92-2637c3ec175b" containerName="cleanup"
Nov 01 04:30:27.373593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373553    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5771967-299c-4e10-bd92-2637c3ec175b" containerName="cleanup"
Nov 01 04:30:27.373593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373572    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5c1a2d6a-5672-4106-9332-59330be63eec" containerName="cleanup"
Nov 01 04:30:27.373593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373578    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1a2d6a-5672-4106-9332-59330be63eec" containerName="cleanup"
Nov 01 04:30:27.373593 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373591    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerName="cleanup"
Nov 01 04:30:27.373774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373596    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerName="cleanup"
Nov 01 04:30:27.373774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373676    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b73bbcd-28bf-401f-97f4-301dd26d4c46" containerName="cleanup"
Nov 01 04:30:27.373774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373685    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6a23a3-2d0f-4083-a42e-3067dbe62be7" containerName="cleanup"
Nov 01 04:30:27.373774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373695    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc456f08-2b1d-4434-ac72-bc5eb9a8ac8c" containerName="cleanup"
Nov 01 04:30:27.373774 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.373704    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1a2d6a-5672-4106-9332-59330be63eec" containerName="cleanup"
Nov 01 04:30:27.376860 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.376843    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:27.416142 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.416117    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:30:27.416280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.416151    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:30:27.428720 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.428697    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"]
Nov 01 04:30:27.434439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.434405    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"]
Nov 01 04:30:27.480681 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.480642    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0f0fd1b-lfhlk\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:27.480681 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.480685    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0f0fd1b-lfhlk\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:27.480880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.480798    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9caabc2-rzb6k\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:27.480880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.480830    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9caabc2-rzb6k\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:27.480880 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.480855    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcjbf\" (UniqueName: \"kubernetes.io/projected/d0039588-fe92-410b-ad5e-3cc921a48f44-kube-api-access-vcjbf\") pod \"maas-api-revoke-sub-e9caabc2-rzb6k\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:27.481010 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.480914    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p2nc\" (UniqueName: \"kubernetes.io/projected/4330d5aa-89dd-4a62-8373-3ca2500825be-kube-api-access-9p2nc\") pod \"maas-api-revoke-sub-a0f0fd1b-lfhlk\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:27.581830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.581737    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9caabc2-rzb6k\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:27.581830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.581779    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcjbf\" (UniqueName: \"kubernetes.io/projected/d0039588-fe92-410b-ad5e-3cc921a48f44-kube-api-access-vcjbf\") pod \"maas-api-revoke-sub-e9caabc2-rzb6k\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:27.581830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.581808    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p2nc\" (UniqueName: \"kubernetes.io/projected/4330d5aa-89dd-4a62-8373-3ca2500825be-kube-api-access-9p2nc\") pod \"maas-api-revoke-sub-a0f0fd1b-lfhlk\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:27.582133 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.581909    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0f0fd1b-lfhlk\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:27.582133 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.581939    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0f0fd1b-lfhlk\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:27.582133 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.582016    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9caabc2-rzb6k\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:27.582676 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.582645    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9caabc2-rzb6k\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:27.582800 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.582742    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0f0fd1b-lfhlk\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:27.584750 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.584715    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0f0fd1b-lfhlk\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:27.584942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.584922    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9caabc2-rzb6k\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:27.638781 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.638749    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p2nc\" (UniqueName: \"kubernetes.io/projected/4330d5aa-89dd-4a62-8373-3ca2500825be-kube-api-access-9p2nc\") pod \"maas-api-revoke-sub-a0f0fd1b-lfhlk\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:27.638938 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.638754    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcjbf\" (UniqueName: \"kubernetes.io/projected/d0039588-fe92-410b-ad5e-3cc921a48f44-kube-api-access-vcjbf\") pod \"maas-api-revoke-sub-e9caabc2-rzb6k\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:27.644478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.644448    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:27.690317 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:27.690285    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:28.015261 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:28.015235    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"]
Nov 01 04:30:28.017456 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:30:28.017404    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4330d5aa_89dd_4a62_8373_3ca2500825be.slice/crio-f61d3a506f36129d9f0c36c2772fb9fe8fa478dfd1751b923b4bd16bd34e2f14 WatchSource:0}: Error finding container f61d3a506f36129d9f0c36c2772fb9fe8fa478dfd1751b923b4bd16bd34e2f14: Status 404 returned error can't find the container with id f61d3a506f36129d9f0c36c2772fb9fe8fa478dfd1751b923b4bd16bd34e2f14
Nov 01 04:30:28.127129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:28.127104    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"]
Nov 01 04:30:28.128336 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:30:28.128307    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0039588_fe92_410b_ad5e_3cc921a48f44.slice/crio-b9c82fb80bb568c1acfb03e253b37b918eddb74db93418b99ec0e9f8cfef6498 WatchSource:0}: Error finding container b9c82fb80bb568c1acfb03e253b37b918eddb74db93418b99ec0e9f8cfef6498: Status 404 returned error can't find the container with id b9c82fb80bb568c1acfb03e253b37b918eddb74db93418b99ec0e9f8cfef6498
Nov 01 04:30:28.438511 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:28.438479    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k" event={"ID":"d0039588-fe92-410b-ad5e-3cc921a48f44","Type":"ContainerStarted","Data":"3446cfe5aa6b615c96adc9f97eb094f2ad9b5f1258cbe34e9c376a3cc97040e0"}
Nov 01 04:30:28.438948 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:28.438539    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k" event={"ID":"d0039588-fe92-410b-ad5e-3cc921a48f44","Type":"ContainerStarted","Data":"b9c82fb80bb568c1acfb03e253b37b918eddb74db93418b99ec0e9f8cfef6498"}
Nov 01 04:30:28.440309 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:28.440283    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk" event={"ID":"4330d5aa-89dd-4a62-8373-3ca2500825be","Type":"ContainerStarted","Data":"5fbb39af66f072b4c89b3b88d66d449107e0cacac060e4e804f226bacf08e684"}
Nov 01 04:30:28.440439 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:28.440322    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk" event={"ID":"4330d5aa-89dd-4a62-8373-3ca2500825be","Type":"ContainerStarted","Data":"f61d3a506f36129d9f0c36c2772fb9fe8fa478dfd1751b923b4bd16bd34e2f14"}
Nov 01 04:30:28.546699 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:28.546588    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k" podStartSLOduration=1.546574502 podStartE2EDuration="1.546574502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:30:28.545964176 +0000 UTC m=+2897.107997204" watchObservedRunningTime="2026-11-01 04:30:28.546574502 +0000 UTC m=+2897.108607523"
Nov 01 04:30:28.654922 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:28.654362    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk" podStartSLOduration=1.6543462039999999 podStartE2EDuration="1.654346204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:30:28.65289176 +0000 UTC m=+2897.214924774" watchObservedRunningTime="2026-11-01 04:30:28.654346204 +0000 UTC m=+2897.216379223"
Nov 01 04:30:28.722188 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:28.722156    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:30:29.448232 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:29.448196    4882 generic.go:356] "Generic (PLEG): container finished" podID="d0039588-fe92-410b-ad5e-3cc921a48f44" containerID="3446cfe5aa6b615c96adc9f97eb094f2ad9b5f1258cbe34e9c376a3cc97040e0" exitCode=0
Nov 01 04:30:29.448758 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:29.448287    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k" event={"ID":"d0039588-fe92-410b-ad5e-3cc921a48f44","Type":"ContainerDied","Data":"3446cfe5aa6b615c96adc9f97eb094f2ad9b5f1258cbe34e9c376a3cc97040e0"}
Nov 01 04:30:29.450084 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:29.450061    4882 generic.go:356] "Generic (PLEG): container finished" podID="4330d5aa-89dd-4a62-8373-3ca2500825be" containerID="5fbb39af66f072b4c89b3b88d66d449107e0cacac060e4e804f226bacf08e684" exitCode=0
Nov 01 04:30:29.450203 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:29.450100    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk" event={"ID":"4330d5aa-89dd-4a62-8373-3ca2500825be","Type":"ContainerDied","Data":"5fbb39af66f072b4c89b3b88d66d449107e0cacac060e4e804f226bacf08e684"}
Nov 01 04:30:30.605080 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.605049    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:30.641364 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.641341    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:30.714359 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.714289    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0039588-fe92-410b-ad5e-3cc921a48f44-kube-api-access-vcjbf\" (UniqueName: \"kubernetes.io/projected/d0039588-fe92-410b-ad5e-3cc921a48f44-kube-api-access-vcjbf\") pod \"d0039588-fe92-410b-ad5e-3cc921a48f44\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") "
Nov 01 04:30:30.714359 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.714334    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-xks-ca\") pod \"4330d5aa-89dd-4a62-8373-3ca2500825be\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") "
Nov 01 04:30:30.714587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.714466    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-xks-ca\") pod \"d0039588-fe92-410b-ad5e-3cc921a48f44\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") "
Nov 01 04:30:30.714652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.714590    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-openshift-ca\") pod \"4330d5aa-89dd-4a62-8373-3ca2500825be\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") "
Nov 01 04:30:30.714652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.714622    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-openshift-ca\") pod \"d0039588-fe92-410b-ad5e-3cc921a48f44\" (UID: \"d0039588-fe92-410b-ad5e-3cc921a48f44\") "
Nov 01 04:30:30.714762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.714659    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4330d5aa-89dd-4a62-8373-3ca2500825be-kube-api-access-9p2nc\" (UniqueName: \"kubernetes.io/projected/4330d5aa-89dd-4a62-8373-3ca2500825be-kube-api-access-9p2nc\") pod \"4330d5aa-89dd-4a62-8373-3ca2500825be\" (UID: \"4330d5aa-89dd-4a62-8373-3ca2500825be\") "
Nov 01 04:30:30.714975 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.714946    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-openshift-ca" pod "4330d5aa-89dd-4a62-8373-3ca2500825be" (UID: "4330d5aa-89dd-4a62-8373-3ca2500825be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:30.715074 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.715056    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:30.715282 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.715255    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-openshift-ca" pod "d0039588-fe92-410b-ad5e-3cc921a48f44" (UID: "d0039588-fe92-410b-ad5e-3cc921a48f44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:30.716749 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.716717    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-xks-ca" pod "4330d5aa-89dd-4a62-8373-3ca2500825be" (UID: "4330d5aa-89dd-4a62-8373-3ca2500825be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:30:30.716749 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.716727    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-xks-ca" pod "d0039588-fe92-410b-ad5e-3cc921a48f44" (UID: "d0039588-fe92-410b-ad5e-3cc921a48f44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:30:30.716927 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.716779    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0039588-fe92-410b-ad5e-3cc921a48f44-kube-api-access-vcjbf" pod "d0039588-fe92-410b-ad5e-3cc921a48f44" (UID: "d0039588-fe92-410b-ad5e-3cc921a48f44"). InnerVolumeSpecName "kube-api-access-vcjbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:30.717045 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.717018    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4330d5aa-89dd-4a62-8373-3ca2500825be-kube-api-access-9p2nc" pod "4330d5aa-89dd-4a62-8373-3ca2500825be" (UID: "4330d5aa-89dd-4a62-8373-3ca2500825be"). InnerVolumeSpecName "kube-api-access-9p2nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:30.815836 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.815801    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcjbf\" (UniqueName: \"kubernetes.io/projected/d0039588-fe92-410b-ad5e-3cc921a48f44-kube-api-access-vcjbf\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:30.815836 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.815830    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4330d5aa-89dd-4a62-8373-3ca2500825be-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:30.815836 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.815842    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:30.816083 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.815853    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0039588-fe92-410b-ad5e-3cc921a48f44-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:30.816083 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:30.815862    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p2nc\" (UniqueName: \"kubernetes.io/projected/4330d5aa-89dd-4a62-8373-3ca2500825be-kube-api-access-9p2nc\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:31.464408 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:31.464376    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"
Nov 01 04:30:31.464606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:31.464383    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk" event={"ID":"4330d5aa-89dd-4a62-8373-3ca2500825be","Type":"ContainerDied","Data":"f61d3a506f36129d9f0c36c2772fb9fe8fa478dfd1751b923b4bd16bd34e2f14"}
Nov 01 04:30:31.464606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:31.464474    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f61d3a506f36129d9f0c36c2772fb9fe8fa478dfd1751b923b4bd16bd34e2f14"
Nov 01 04:30:31.465789 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:31.465767    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"
Nov 01 04:30:31.465789 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:31.465779    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k" event={"ID":"d0039588-fe92-410b-ad5e-3cc921a48f44","Type":"ContainerDied","Data":"b9c82fb80bb568c1acfb03e253b37b918eddb74db93418b99ec0e9f8cfef6498"}
Nov 01 04:30:31.465962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:31.465833    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9c82fb80bb568c1acfb03e253b37b918eddb74db93418b99ec0e9f8cfef6498"
Nov 01 04:30:38.672870 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.672835    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"]
Nov 01 04:30:38.673301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.673288    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d0039588-fe92-410b-ad5e-3cc921a48f44" containerName="revoke-keys"
Nov 01 04:30:38.673362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.673302    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0039588-fe92-410b-ad5e-3cc921a48f44" containerName="revoke-keys"
Nov 01 04:30:38.673362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.673346    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4330d5aa-89dd-4a62-8373-3ca2500825be" containerName="revoke-keys"
Nov 01 04:30:38.673362 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.673351    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4330d5aa-89dd-4a62-8373-3ca2500825be" containerName="revoke-keys"
Nov 01 04:30:38.673490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.673423    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4330d5aa-89dd-4a62-8373-3ca2500825be" containerName="revoke-keys"
Nov 01 04:30:38.673490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.673448    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0039588-fe92-410b-ad5e-3cc921a48f44" containerName="revoke-keys"
Nov 01 04:30:38.676694 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.676674    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:38.770587 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.770548    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:30:38.770762 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.770649    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:30:38.773769 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.773746    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"]
Nov 01 04:30:38.806306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.806276    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bd2q\" (UniqueName: \"kubernetes.io/projected/e52816b2-9171-4f14-a409-e5316862c985-kube-api-access-8bd2q\") pod \"maas-api-revoke-sub-bff8b8d9-rh28n\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:38.806461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.806326    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52816b2-9171-4f14-a409-e5316862c985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bff8b8d9-rh28n\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:38.806461 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.806393    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52816b2-9171-4f14-a409-e5316862c985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bff8b8d9-rh28n\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:38.907863 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.907826    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bd2q\" (UniqueName: \"kubernetes.io/projected/e52816b2-9171-4f14-a409-e5316862c985-kube-api-access-8bd2q\") pod \"maas-api-revoke-sub-bff8b8d9-rh28n\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:38.908019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.907891    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52816b2-9171-4f14-a409-e5316862c985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bff8b8d9-rh28n\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:38.908019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.907923    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52816b2-9171-4f14-a409-e5316862c985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bff8b8d9-rh28n\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:38.908619 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.908589    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52816b2-9171-4f14-a409-e5316862c985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bff8b8d9-rh28n\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:38.910272 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.910249    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52816b2-9171-4f14-a409-e5316862c985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bff8b8d9-rh28n\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:38.966245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.966225    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bd2q\" (UniqueName: \"kubernetes.io/projected/e52816b2-9171-4f14-a409-e5316862c985-kube-api-access-8bd2q\") pod \"maas-api-revoke-sub-bff8b8d9-rh28n\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:38.989705 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:38.989682    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:39.359928 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:39.359906    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"]
Nov 01 04:30:39.362214 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:30:39.362176    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52816b2_9171_4f14_a409_e5316862c985.slice/crio-f9773d6332169001417911a3a4dd8d186fd3abeee1c5cfb11d68bf609cd62b2a WatchSource:0}: Error finding container f9773d6332169001417911a3a4dd8d186fd3abeee1c5cfb11d68bf609cd62b2a: Status 404 returned error can't find the container with id f9773d6332169001417911a3a4dd8d186fd3abeee1c5cfb11d68bf609cd62b2a
Nov 01 04:30:39.522099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:39.522063    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n" event={"ID":"e52816b2-9171-4f14-a409-e5316862c985","Type":"ContainerStarted","Data":"f9773d6332169001417911a3a4dd8d186fd3abeee1c5cfb11d68bf609cd62b2a"}
Nov 01 04:30:40.530202 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:40.530102    4882 generic.go:356] "Generic (PLEG): container finished" podID="e52816b2-9171-4f14-a409-e5316862c985" containerID="7cb03490de7de4a82e9f2fabc7a3b5ba5c30a801450f1d760cfd3febafe97664" exitCode=0
Nov 01 04:30:40.530742 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:40.530198    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n" event={"ID":"e52816b2-9171-4f14-a409-e5316862c985","Type":"ContainerDied","Data":"7cb03490de7de4a82e9f2fabc7a3b5ba5c30a801450f1d760cfd3febafe97664"}
Nov 01 04:30:41.709279 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:41.709257    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:41.838491 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:41.838356    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e52816b2-9171-4f14-a409-e5316862c985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52816b2-9171-4f14-a409-e5316862c985-maas-api-xks-ca\") pod \"e52816b2-9171-4f14-a409-e5316862c985\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") "
Nov 01 04:30:41.838756 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:41.838722    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52816b2-9171-4f14-a409-e5316862c985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52816b2-9171-4f14-a409-e5316862c985-maas-api-openshift-ca\") pod \"e52816b2-9171-4f14-a409-e5316862c985\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") "
Nov 01 04:30:41.838924 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:41.838905    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52816b2-9171-4f14-a409-e5316862c985-kube-api-access-8bd2q\" (UniqueName: \"kubernetes.io/projected/e52816b2-9171-4f14-a409-e5316862c985-kube-api-access-8bd2q\") pod \"e52816b2-9171-4f14-a409-e5316862c985\" (UID: \"e52816b2-9171-4f14-a409-e5316862c985\") "
Nov 01 04:30:41.839213 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:41.839185    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52816b2-9171-4f14-a409-e5316862c985-maas-api-openshift-ca" pod "e52816b2-9171-4f14-a409-e5316862c985" (UID: "e52816b2-9171-4f14-a409-e5316862c985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:41.840671 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:41.840645    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e52816b2-9171-4f14-a409-e5316862c985-maas-api-xks-ca" pod "e52816b2-9171-4f14-a409-e5316862c985" (UID: "e52816b2-9171-4f14-a409-e5316862c985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:30:41.841837 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:41.841810    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52816b2-9171-4f14-a409-e5316862c985-kube-api-access-8bd2q" pod "e52816b2-9171-4f14-a409-e5316862c985" (UID: "e52816b2-9171-4f14-a409-e5316862c985"). InnerVolumeSpecName "kube-api-access-8bd2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:41.941277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:41.941237    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e52816b2-9171-4f14-a409-e5316862c985-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:41.941277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:41.941280    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e52816b2-9171-4f14-a409-e5316862c985-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:41.941506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:41.941299    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bd2q\" (UniqueName: \"kubernetes.io/projected/e52816b2-9171-4f14-a409-e5316862c985-kube-api-access-8bd2q\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:42.544666 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:42.544631    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n"
Nov 01 04:30:42.544845 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:42.544669    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bff8b8d9-rh28n" event={"ID":"e52816b2-9171-4f14-a409-e5316862c985","Type":"ContainerDied","Data":"f9773d6332169001417911a3a4dd8d186fd3abeee1c5cfb11d68bf609cd62b2a"}
Nov 01 04:30:42.544845 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:42.544724    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9773d6332169001417911a3a4dd8d186fd3abeee1c5cfb11d68bf609cd62b2a"
Nov 01 04:30:46.234673 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:46.234637    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"]
Nov 01 04:30:46.235166 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:46.234888    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy" containerID="cri-o://fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e" gracePeriod=30
Nov 01 04:30:47.270202 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.270161    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"]
Nov 01 04:30:47.270854 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.270830    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e52816b2-9171-4f14-a409-e5316862c985" containerName="revoke-keys"
Nov 01 04:30:47.270951 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.270855    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52816b2-9171-4f14-a409-e5316862c985" containerName="revoke-keys"
Nov 01 04:30:47.271074 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.271061    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52816b2-9171-4f14-a409-e5316862c985" containerName="revoke-keys"
Nov 01 04:30:47.274851 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.274829    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:47.303592 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.303523    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:47.303892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.303658    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:47.303892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.303710    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bft47\" (UniqueName: \"kubernetes.io/projected/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-kube-api-access-bft47\") pod \"maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:47.323301 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.323280    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:30:47.369089 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.369054    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:30:47.374043 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.374008    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"]
Nov 01 04:30:47.404805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.404770    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:47.404967 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.404853    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:47.404967 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.404889    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bft47\" (UniqueName: \"kubernetes.io/projected/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-kube-api-access-bft47\") pod \"maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:47.405609 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.405591    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:47.407200 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.407172    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:47.461747 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.461715    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bft47\" (UniqueName: \"kubernetes.io/projected/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-kube-api-access-bft47\") pod \"maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:47.589293 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.589214    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:47.964500 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:47.964472    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"]
Nov 01 04:30:47.966454 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:30:47.966402    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03f1b0b_70aa_4d41_a311_8e8c0f8097b6.slice/crio-aca4b396559d496ee9ad7355fc1eb92070d49a17060ce1d5f4acffb19cb241bc WatchSource:0}: Error finding container aca4b396559d496ee9ad7355fc1eb92070d49a17060ce1d5f4acffb19cb241bc: Status 404 returned error can't find the container with id aca4b396559d496ee9ad7355fc1eb92070d49a17060ce1d5f4acffb19cb241bc
Nov 01 04:30:48.593505 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:48.593403    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx" event={"ID":"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6","Type":"ContainerStarted","Data":"d35eceb89ff0902f9d4593c1776c2e7a30ec296804d09d6e66f515212082168a"}
Nov 01 04:30:48.593505 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:48.593475    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx" event={"ID":"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6","Type":"ContainerStarted","Data":"aca4b396559d496ee9ad7355fc1eb92070d49a17060ce1d5f4acffb19cb241bc"}
Nov 01 04:30:48.702210 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:48.702149    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx" podStartSLOduration=1.702122441 podStartE2EDuration="1.702122441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:30:48.700189347 +0000 UTC m=+2917.262222366" watchObservedRunningTime="2026-11-01 04:30:48.702122441 +0000 UTC m=+2917.264155510"
Nov 01 04:30:49.602506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:49.602475    4882 generic.go:356] "Generic (PLEG): container finished" podID="f03f1b0b-70aa-4d41-a311-8e8c0f8097b6" containerID="d35eceb89ff0902f9d4593c1776c2e7a30ec296804d09d6e66f515212082168a" exitCode=0
Nov 01 04:30:49.602887 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:49.602548    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx" event={"ID":"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6","Type":"ContainerDied","Data":"d35eceb89ff0902f9d4593c1776c2e7a30ec296804d09d6e66f515212082168a"}
Nov 01 04:30:50.750330 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:50.750305    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:50.841206 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:50.841167    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-xks-ca\") pod \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") "
Nov 01 04:30:50.841358 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:50.841232    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-kube-api-access-bft47\" (UniqueName: \"kubernetes.io/projected/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-kube-api-access-bft47\") pod \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") "
Nov 01 04:30:50.841358 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:50.841297    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-openshift-ca\") pod \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\" (UID: \"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6\") "
Nov 01 04:30:50.841733 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:50.841705    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-openshift-ca" pod "f03f1b0b-70aa-4d41-a311-8e8c0f8097b6" (UID: "f03f1b0b-70aa-4d41-a311-8e8c0f8097b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:50.843329 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:50.843303    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-kube-api-access-bft47" pod "f03f1b0b-70aa-4d41-a311-8e8c0f8097b6" (UID: "f03f1b0b-70aa-4d41-a311-8e8c0f8097b6"). InnerVolumeSpecName "kube-api-access-bft47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:50.843329 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:50.843319    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-xks-ca" pod "f03f1b0b-70aa-4d41-a311-8e8c0f8097b6" (UID: "f03f1b0b-70aa-4d41-a311-8e8c0f8097b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:30:50.942793 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:50.942760    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bft47\" (UniqueName: \"kubernetes.io/projected/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-kube-api-access-bft47\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:50.942793 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:50.942788    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:50.942793 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:50.942799    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:51.481600 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.481574    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:30:51.549680 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.549609    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91fbb559-853d-47f1-a46a-347db176506c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fbb559-853d-47f1-a46a-347db176506c-istiod-ca-cert\") pod \"91fbb559-853d-47f1-a46a-347db176506c\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") "
Nov 01 04:30:51.549680 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.549658    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91fbb559-853d-47f1-a46a-347db176506c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fbb559-853d-47f1-a46a-347db176506c-istio-podinfo\") pod \"91fbb559-853d-47f1-a46a-347db176506c\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") "
Nov 01 04:30:51.549859 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.549722    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-certs\") pod \"91fbb559-853d-47f1-a46a-347db176506c\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") "
Nov 01 04:30:51.549859 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.549759    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-credential-socket\") pod \"91fbb559-853d-47f1-a46a-347db176506c\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") "
Nov 01 04:30:51.549859 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.549791    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-socket\") pod \"91fbb559-853d-47f1-a46a-347db176506c\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") "
Nov 01 04:30:51.549997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.549869    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-data\") pod \"91fbb559-853d-47f1-a46a-347db176506c\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") "
Nov 01 04:30:51.549997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.549892    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-envoy\") pod \"91fbb559-853d-47f1-a46a-347db176506c\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") "
Nov 01 04:30:51.549997 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.549962    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-kube-api-access-vx75s\" (UniqueName: \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-kube-api-access-vx75s\") pod \"91fbb559-853d-47f1-a46a-347db176506c\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") "
Nov 01 04:30:51.550160 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.550015    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-istio-token\" (UniqueName: \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-istio-token\") pod \"91fbb559-853d-47f1-a46a-347db176506c\" (UID: \"91fbb559-853d-47f1-a46a-347db176506c\") "
Nov 01 04:30:51.550277 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.550011    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-certs" pod "91fbb559-853d-47f1-a46a-347db176506c" (UID: "91fbb559-853d-47f1-a46a-347db176506c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:51.550350 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.550026    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fbb559-853d-47f1-a46a-347db176506c-istiod-ca-cert" pod "91fbb559-853d-47f1-a46a-347db176506c" (UID: "91fbb559-853d-47f1-a46a-347db176506c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:30:51.550350 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.550129    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-credential-socket" pod "91fbb559-853d-47f1-a46a-347db176506c" (UID: "91fbb559-853d-47f1-a46a-347db176506c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:51.550350 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.550190    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-data" pod "91fbb559-853d-47f1-a46a-347db176506c" (UID: "91fbb559-853d-47f1-a46a-347db176506c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:51.550350 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.550208    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-socket" pod "91fbb559-853d-47f1-a46a-347db176506c" (UID: "91fbb559-853d-47f1-a46a-347db176506c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:51.550571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.550498    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:51.550571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.550519    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:51.550571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.550535    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:51.550571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.550550    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:51.550571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.550563    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fbb559-853d-47f1-a46a-347db176506c-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:51.551901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.551874    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91fbb559-853d-47f1-a46a-347db176506c-istio-podinfo" pod "91fbb559-853d-47f1-a46a-347db176506c" (UID: "91fbb559-853d-47f1-a46a-347db176506c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:30:51.552226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.552205    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-kube-api-access-vx75s" pod "91fbb559-853d-47f1-a46a-347db176506c" (UID: "91fbb559-853d-47f1-a46a-347db176506c"). InnerVolumeSpecName "kube-api-access-vx75s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:51.552561 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.552541    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-envoy" pod "91fbb559-853d-47f1-a46a-347db176506c" (UID: "91fbb559-853d-47f1-a46a-347db176506c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:30:51.552633 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.552541    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-istio-token" pod "91fbb559-853d-47f1-a46a-347db176506c" (UID: "91fbb559-853d-47f1-a46a-347db176506c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:30:51.615684 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.615664    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"
Nov 01 04:30:51.615827 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.615801    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx" event={"ID":"f03f1b0b-70aa-4d41-a311-8e8c0f8097b6","Type":"ContainerDied","Data":"aca4b396559d496ee9ad7355fc1eb92070d49a17060ce1d5f4acffb19cb241bc"}
Nov 01 04:30:51.615936 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.615851    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aca4b396559d496ee9ad7355fc1eb92070d49a17060ce1d5f4acffb19cb241bc"
Nov 01 04:30:51.617684 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.617661    4882 generic.go:356] "Generic (PLEG): container finished" podID="91fbb559-853d-47f1-a46a-347db176506c" containerID="fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e" exitCode=0
Nov 01 04:30:51.617788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.617705    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"
Nov 01 04:30:51.617788 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.617742    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" event={"ID":"91fbb559-853d-47f1-a46a-347db176506c","Type":"ContainerDied","Data":"fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e"}
Nov 01 04:30:51.617919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.617794    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k" event={"ID":"91fbb559-853d-47f1-a46a-347db176506c","Type":"ContainerDied","Data":"9c1e6e03fc75b5a272c057c696d999b907d75c2ef2ade6a8d76f5e6f2d6cd89f"}
Nov 01 04:30:51.617919 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.617831    4882 scope.go:122] "RemoveContainer" containerID="fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e"
Nov 01 04:30:51.627377 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.627360    4882 scope.go:122] "RemoveContainer" containerID="81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9"
Nov 01 04:30:51.636056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.636035    4882 scope.go:122] "RemoveContainer" containerID="fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e"
Nov 01 04:30:51.636318 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:51.636300    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e\": container with ID starting with fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e not found: ID does not exist" containerID="fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e"
Nov 01 04:30:51.636384 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.636325    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e"} err="failed to get container status \"fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e\": rpc error: code = NotFound desc = could not find container \"fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e\": container with ID starting with fadf69cbe5c928ab887a59896e5580548cb6330f37b86a955756a2e17798068e not found: ID does not exist"
Nov 01 04:30:51.636384 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.636342    4882 scope.go:122] "RemoveContainer" containerID="81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9"
Nov 01 04:30:51.636617 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:51.636601    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9\": container with ID starting with 81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9 not found: ID does not exist" containerID="81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9"
Nov 01 04:30:51.636674 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.636621    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9"} err="failed to get container status \"81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9\": rpc error: code = NotFound desc = could not find container \"81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9\": container with ID starting with 81b92c2ca4b509c77f53dc5f5e1b6240d39d3bd110bf0decefc0346a8e7f15e9 not found: ID does not exist"
Nov 01 04:30:51.652016 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.651998    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fbb559-853d-47f1-a46a-347db176506c-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:51.652111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.652017    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx75s\" (UniqueName: \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-kube-api-access-vx75s\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:51.652111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.652030    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fbb559-853d-47f1-a46a-347db176506c-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:51.652111 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.652039    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fbb559-853d-47f1-a46a-347db176506c-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:30:51.695394 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.695368    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"]
Nov 01 04:30:51.799189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.799157    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"]
Nov 01 04:30:51.803199 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:51.803129    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b8319b-openshift-default-567b77d49-6zv5k"]
Nov 01 04:30:53.599345 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:30:53.599310    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91fbb559-853d-47f1-a46a-347db176506c" path="/var/lib/kubelet/pods/91fbb559-853d-47f1-a46a-347db176506c/volumes"
Nov 01 04:30:58.723210 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:30:58.723169    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:31:06.715734 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:06.715691    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" containerID="cri-o://96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b" gracePeriod=45
Nov 01 04:31:06.736651 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:06.736619    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": dial tcp 10.132.0.71:8000: connect: connection refused"
Nov 01 04:31:06.979369 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:06.979307    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 04:31:07.128644 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.128608    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b62aaf-4775-4a62-a49c-c2ada60bea08-tls-certs\" (UniqueName: \"kubernetes.io/secret/28b62aaf-4775-4a62-a49c-c2ada60bea08-tls-certs\") pod \"28b62aaf-4775-4a62-a49c-c2ada60bea08\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") "
Nov 01 04:31:07.128809 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.128682    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-tmp-dir\") pod \"28b62aaf-4775-4a62-a49c-c2ada60bea08\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") "
Nov 01 04:31:07.128809 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.128775    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-model-cache\") pod \"28b62aaf-4775-4a62-a49c-c2ada60bea08\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") "
Nov 01 04:31:07.128962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.128872    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-dshm\") pod \"28b62aaf-4775-4a62-a49c-c2ada60bea08\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") "
Nov 01 04:31:07.128962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.128928    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b62aaf-4775-4a62-a49c-c2ada60bea08-kube-api-access-6rxp7\" (UniqueName: \"kubernetes.io/projected/28b62aaf-4775-4a62-a49c-c2ada60bea08-kube-api-access-6rxp7\") pod \"28b62aaf-4775-4a62-a49c-c2ada60bea08\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") "
Nov 01 04:31:07.129085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.128990    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-home\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-home\") pod \"28b62aaf-4775-4a62-a49c-c2ada60bea08\" (UID: \"28b62aaf-4775-4a62-a49c-c2ada60bea08\") "
Nov 01 04:31:07.129085 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.129057    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-model-cache" pod "28b62aaf-4775-4a62-a49c-c2ada60bea08" (UID: "28b62aaf-4775-4a62-a49c-c2ada60bea08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:31:07.129481 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.129455    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-model-cache\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:07.129656 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.129634    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-home" pod "28b62aaf-4775-4a62-a49c-c2ada60bea08" (UID: "28b62aaf-4775-4a62-a49c-c2ada60bea08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:31:07.129776 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.129347    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-tmp-dir" pod "28b62aaf-4775-4a62-a49c-c2ada60bea08" (UID: "28b62aaf-4775-4a62-a49c-c2ada60bea08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:31:07.131227 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.131203    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-dshm" pod "28b62aaf-4775-4a62-a49c-c2ada60bea08" (UID: "28b62aaf-4775-4a62-a49c-c2ada60bea08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:31:07.131356 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.131339    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b62aaf-4775-4a62-a49c-c2ada60bea08-tls-certs" pod "28b62aaf-4775-4a62-a49c-c2ada60bea08" (UID: "28b62aaf-4775-4a62-a49c-c2ada60bea08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:31:07.131536 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.131514    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b62aaf-4775-4a62-a49c-c2ada60bea08-kube-api-access-6rxp7" pod "28b62aaf-4775-4a62-a49c-c2ada60bea08" (UID: "28b62aaf-4775-4a62-a49c-c2ada60bea08"). InnerVolumeSpecName "kube-api-access-6rxp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:31:07.231099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.231013    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-dshm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:07.231099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.231046    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rxp7\" (UniqueName: \"kubernetes.io/projected/28b62aaf-4775-4a62-a49c-c2ada60bea08-kube-api-access-6rxp7\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:07.231099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.231057    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-home\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:07.231099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.231065    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b62aaf-4775-4a62-a49c-c2ada60bea08-tls-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:07.231099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.231074    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b62aaf-4775-4a62-a49c-c2ada60bea08-tmp-dir\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:07.723909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.723872    4882 generic.go:356] "Generic (PLEG): container finished" podID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerID="96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b" exitCode=0
Nov 01 04:31:07.724308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.723914    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"
Nov 01 04:31:07.724308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.723931    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" event={"ID":"28b62aaf-4775-4a62-a49c-c2ada60bea08","Type":"ContainerDied","Data":"96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b"}
Nov 01 04:31:07.724308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.723964    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh" event={"ID":"28b62aaf-4775-4a62-a49c-c2ada60bea08","Type":"ContainerDied","Data":"e701b70041cb4302e239f1dbac6ab17e8a353f18abf9c396257546f15db24617"}
Nov 01 04:31:07.724308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.723981    4882 scope.go:122] "RemoveContainer" containerID="96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b"
Nov 01 04:31:07.733550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.733529    4882 scope.go:122] "RemoveContainer" containerID="96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b"
Nov 01 04:31:07.733820 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:31:07.733789    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b\": container with ID starting with 96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b not found: ID does not exist" containerID="96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b"
Nov 01 04:31:07.733875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.733828    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b"} err="failed to get container status \"96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b\": rpc error: code = NotFound desc = could not find container \"96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b\": container with ID starting with 96d183b63d361ca3966c05c2d47fbd591735312236dcc3e8e48334111c3da70b not found: ID does not exist"
Nov 01 04:31:07.876375 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.876302    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"]
Nov 01 04:31:07.880437 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:07.880403    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/simulator-w4-a9c5b3-kserve-75df979996-jspgh"]
Nov 01 04:31:09.600137 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:09.600107    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" path="/var/lib/kubelet/pods/28b62aaf-4775-4a62-a49c-c2ada60bea08/volumes"
Nov 01 04:31:13.333908 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:13.333875    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"]
Nov 01 04:31:13.379571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:13.379536    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ad9e22-2tdd2"]
Nov 01 04:31:13.600421 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:13.600345    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb84c12-e9cf-4428-b367-e617b91f4c03" path="/var/lib/kubelet/pods/8bb84c12-e9cf-4428-b367-e617b91f4c03/volumes"
Nov 01 04:31:13.827099 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:13.827067    4882 scope.go:122] "RemoveContainer" containerID="b47d770afcb216ee2becbbe04f6a4517f9c6802bb9bdda98b96ad4886f308ed8"
Nov 01 04:31:13.836495 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:13.836470    4882 scope.go:122] "RemoveContainer" containerID="7095fec4eb5a6c0a0888f76b0475e03bfc64ec0868935a5476d5dc85317bf62c"
Nov 01 04:31:27.055146 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.055103    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"]
Nov 01 04:31:27.100830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.100793    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"]
Nov 01 04:31:27.103136 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.103109    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e113b8f-t75m8"]
Nov 01 04:31:27.105136 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.105117    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7736ecb-zs48z"]
Nov 01 04:31:27.598970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.598939    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a9b81c4-51e3-4cfe-bcee-673bc419e77d" path="/var/lib/kubelet/pods/4a9b81c4-51e3-4cfe-bcee-673bc419e77d/volumes"
Nov 01 04:31:27.599349 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.599335    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995" path="/var/lib/kubelet/pods/533b9cfb-1cc1-4a4f-8cdd-f48d03e0e995/volumes"
Nov 01 04:31:27.697830 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.697795    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"]
Nov 01 04:31:27.698263 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.698250    4882 cpu_manager.go:414] "containerMap: removing container" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy"
Nov 01 04:31:27.698313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.698264    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy"
Nov 01 04:31:27.698313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.698297    4882 cpu_manager.go:414] "containerMap: removing container" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main"
Nov 01 04:31:27.698313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.698302    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main"
Nov 01 04:31:27.698313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.698310    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f03f1b0b-70aa-4d41-a311-8e8c0f8097b6" containerName="revoke-keys"
Nov 01 04:31:27.698455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.698315    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03f1b0b-70aa-4d41-a311-8e8c0f8097b6" containerName="revoke-keys"
Nov 01 04:31:27.698455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.698412    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b62aaf-4775-4a62-a49c-c2ada60bea08" containerName="main"
Nov 01 04:31:27.698455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.698439    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03f1b0b-70aa-4d41-a311-8e8c0f8097b6" containerName="revoke-keys"
Nov 01 04:31:27.698455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.698449    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy"
Nov 01 04:31:27.698592 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.698463    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy"
Nov 01 04:31:27.703833 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.703810    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:27.756208 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.756178    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"]
Nov 01 04:31:27.756686 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.756672    4882 cpu_manager.go:414] "containerMap: removing container" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy"
Nov 01 04:31:27.756744 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.756689    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fbb559-853d-47f1-a46a-347db176506c" containerName="istio-proxy"
Nov 01 04:31:27.760051 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.760033    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:27.799340 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.799309    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:31:27.799540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.799518    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:31:27.802552 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.802524    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"]
Nov 01 04:31:27.829960 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.829928    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d584266a-tvbpj\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:27.830117 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.829968    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0932abfc-xzj5f\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:27.830117 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.830004    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d584266a-tvbpj\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:27.830195 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.830122    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0932abfc-xzj5f\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:27.830195 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.830142    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjg6\" (UniqueName: \"kubernetes.io/projected/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-kube-api-access-5bjg6\") pod \"maas-api-revoke-sub-d584266a-tvbpj\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:27.830280 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.830267    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knzb5\" (UniqueName: \"kubernetes.io/projected/a3b071e4-568c-40a8-9a9f-1faa2168aadb-kube-api-access-knzb5\") pod \"maas-api-revoke-sub-0932abfc-xzj5f\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:27.858166 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.858080    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"]
Nov 01 04:31:27.930904 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.930860    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knzb5\" (UniqueName: \"kubernetes.io/projected/a3b071e4-568c-40a8-9a9f-1faa2168aadb-kube-api-access-knzb5\") pod \"maas-api-revoke-sub-0932abfc-xzj5f\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:27.931077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.930947    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d584266a-tvbpj\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:27.931077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.930969    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0932abfc-xzj5f\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:27.931077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.931001    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d584266a-tvbpj\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:27.931077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.931022    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0932abfc-xzj5f\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:27.931077 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.931047    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjg6\" (UniqueName: \"kubernetes.io/projected/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-kube-api-access-5bjg6\") pod \"maas-api-revoke-sub-d584266a-tvbpj\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:27.931726 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.931700    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d584266a-tvbpj\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:27.931826 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.931756    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0932abfc-xzj5f\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:27.933344 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.933323    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d584266a-tvbpj\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:27.933422 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.933368    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0932abfc-xzj5f\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:27.988124 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.988084    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjg6\" (UniqueName: \"kubernetes.io/projected/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-kube-api-access-5bjg6\") pod \"maas-api-revoke-sub-d584266a-tvbpj\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:27.988281 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:27.988087    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knzb5\" (UniqueName: \"kubernetes.io/projected/a3b071e4-568c-40a8-9a9f-1faa2168aadb-kube-api-access-knzb5\") pod \"maas-api-revoke-sub-0932abfc-xzj5f\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:28.016630 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:28.016606    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:28.073766 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:28.073741    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:28.387129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:28.386991    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"]
Nov 01 04:31:28.491225 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:28.491201    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"]
Nov 01 04:31:28.493417 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:31:28.493376    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4811a7_193b_4797_bdcb_7a5f7e22ee63.slice/crio-756d6ee2b191bc97c26abc712212e99a483abc1c9472c096350688d59af28891 WatchSource:0}: Error finding container 756d6ee2b191bc97c26abc712212e99a483abc1c9472c096350688d59af28891: Status 404 returned error can't find the container with id 756d6ee2b191bc97c26abc712212e99a483abc1c9472c096350688d59af28891
Nov 01 04:31:28.723151 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:31:28.723113    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:31:28.854717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:28.854675    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj" event={"ID":"ac4811a7-193b-4797-bdcb-7a5f7e22ee63","Type":"ContainerStarted","Data":"d72e2130cdc672f2ba6120257b2f5fd254f95f26209668d1444074c87e12cabe"}
Nov 01 04:31:28.854942 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:28.854740    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj" event={"ID":"ac4811a7-193b-4797-bdcb-7a5f7e22ee63","Type":"ContainerStarted","Data":"756d6ee2b191bc97c26abc712212e99a483abc1c9472c096350688d59af28891"}
Nov 01 04:31:28.856363 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:28.856331    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f" event={"ID":"a3b071e4-568c-40a8-9a9f-1faa2168aadb","Type":"ContainerStarted","Data":"e159290b8b6f7188babc618cf9af19b5d7ca118a95132713892dbd65e1653e40"}
Nov 01 04:31:28.856497 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:28.856382    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f" event={"ID":"a3b071e4-568c-40a8-9a9f-1faa2168aadb","Type":"ContainerStarted","Data":"ff8dfac3d7a0e5540dd5c94987930c476a694d78cd6a1c40f0f86cbfc2cd897d"}
Nov 01 04:31:28.965022 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:28.964965    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj" podStartSLOduration=1.964949466 podStartE2EDuration="1.964949466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:31:28.96182539 +0000 UTC m=+2957.523858404" watchObservedRunningTime="2026-11-01 04:31:28.964949466 +0000 UTC m=+2957.526982524"
Nov 01 04:31:29.070829 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:29.070776    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f" podStartSLOduration=2.070762356 podStartE2EDuration="2.070762356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:31:29.068109455 +0000 UTC m=+2957.630142473" watchObservedRunningTime="2026-11-01 04:31:29.070762356 +0000 UTC m=+2957.632795373"
Nov 01 04:31:29.865366 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:29.865328    4882 generic.go:356] "Generic (PLEG): container finished" podID="ac4811a7-193b-4797-bdcb-7a5f7e22ee63" containerID="d72e2130cdc672f2ba6120257b2f5fd254f95f26209668d1444074c87e12cabe" exitCode=0
Nov 01 04:31:29.865849 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:29.865403    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj" event={"ID":"ac4811a7-193b-4797-bdcb-7a5f7e22ee63","Type":"ContainerDied","Data":"d72e2130cdc672f2ba6120257b2f5fd254f95f26209668d1444074c87e12cabe"}
Nov 01 04:31:29.867174 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:29.867147    4882 generic.go:356] "Generic (PLEG): container finished" podID="a3b071e4-568c-40a8-9a9f-1faa2168aadb" containerID="e159290b8b6f7188babc618cf9af19b5d7ca118a95132713892dbd65e1653e40" exitCode=0
Nov 01 04:31:29.867333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:29.867230    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f" event={"ID":"a3b071e4-568c-40a8-9a9f-1faa2168aadb","Type":"ContainerDied","Data":"e159290b8b6f7188babc618cf9af19b5d7ca118a95132713892dbd65e1653e40"}
Nov 01 04:31:31.050555 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.050529    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:31.053714 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.053692    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:31.167265 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.167185    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-xks-ca\") pod \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") "
Nov 01 04:31:31.167457 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.167267    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-kube-api-access-5bjg6\" (UniqueName: \"kubernetes.io/projected/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-kube-api-access-5bjg6\") pod \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") "
Nov 01 04:31:31.167457 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.167334    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-openshift-ca\") pod \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") "
Nov 01 04:31:31.167574 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.167453    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-openshift-ca\") pod \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") "
Nov 01 04:31:31.167574 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.167484    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b071e4-568c-40a8-9a9f-1faa2168aadb-kube-api-access-knzb5\" (UniqueName: \"kubernetes.io/projected/a3b071e4-568c-40a8-9a9f-1faa2168aadb-kube-api-access-knzb5\") pod \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\" (UID: \"a3b071e4-568c-40a8-9a9f-1faa2168aadb\") "
Nov 01 04:31:31.167722 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.167587    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-xks-ca\") pod \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\" (UID: \"ac4811a7-193b-4797-bdcb-7a5f7e22ee63\") "
Nov 01 04:31:31.171473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.167950    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-openshift-ca" pod "a3b071e4-568c-40a8-9a9f-1faa2168aadb" (UID: "a3b071e4-568c-40a8-9a9f-1faa2168aadb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:31:31.171473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.168329    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-openshift-ca" pod "ac4811a7-193b-4797-bdcb-7a5f7e22ee63" (UID: "ac4811a7-193b-4797-bdcb-7a5f7e22ee63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:31:31.171473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.168633    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:31.171473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.168653    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:31.171473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.169728    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-xks-ca" pod "a3b071e4-568c-40a8-9a9f-1faa2168aadb" (UID: "a3b071e4-568c-40a8-9a9f-1faa2168aadb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:31:31.171473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.170120    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-xks-ca" pod "ac4811a7-193b-4797-bdcb-7a5f7e22ee63" (UID: "ac4811a7-193b-4797-bdcb-7a5f7e22ee63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:31:31.171473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.170278    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-kube-api-access-5bjg6" pod "ac4811a7-193b-4797-bdcb-7a5f7e22ee63" (UID: "ac4811a7-193b-4797-bdcb-7a5f7e22ee63"). InnerVolumeSpecName "kube-api-access-5bjg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:31:31.171473 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.170759    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b071e4-568c-40a8-9a9f-1faa2168aadb-kube-api-access-knzb5" pod "a3b071e4-568c-40a8-9a9f-1faa2168aadb" (UID: "a3b071e4-568c-40a8-9a9f-1faa2168aadb"). InnerVolumeSpecName "kube-api-access-knzb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:31:31.270141 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.270106    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knzb5\" (UniqueName: \"kubernetes.io/projected/a3b071e4-568c-40a8-9a9f-1faa2168aadb-kube-api-access-knzb5\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:31.270141 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.270138    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:31.270378 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.270152    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b071e4-568c-40a8-9a9f-1faa2168aadb-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:31.270378 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.270167    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bjg6\" (UniqueName: \"kubernetes.io/projected/ac4811a7-193b-4797-bdcb-7a5f7e22ee63-kube-api-access-5bjg6\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:31:31.880622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.880544    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj"
Nov 01 04:31:31.880622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.880568    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d584266a-tvbpj" event={"ID":"ac4811a7-193b-4797-bdcb-7a5f7e22ee63","Type":"ContainerDied","Data":"756d6ee2b191bc97c26abc712212e99a483abc1c9472c096350688d59af28891"}
Nov 01 04:31:31.880622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.880616    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="756d6ee2b191bc97c26abc712212e99a483abc1c9472c096350688d59af28891"
Nov 01 04:31:31.881991 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.881968    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f" event={"ID":"a3b071e4-568c-40a8-9a9f-1faa2168aadb","Type":"ContainerDied","Data":"ff8dfac3d7a0e5540dd5c94987930c476a694d78cd6a1c40f0f86cbfc2cd897d"}
Nov 01 04:31:31.882117 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.882004    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0932abfc-xzj5f"
Nov 01 04:31:31.882117 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:31.882010    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8dfac3d7a0e5540dd5c94987930c476a694d78cd6a1c40f0f86cbfc2cd897d"
Nov 01 04:31:33.047223 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:33.047189    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"]
Nov 01 04:31:33.093254 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:33.093221    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fc48b4-mpdsj"]
Nov 01 04:31:33.095676 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:33.095652    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"]
Nov 01 04:31:33.097712 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:33.097696    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268bc2e6-dnvjn"]
Nov 01 04:31:33.600112 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:33.600076    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe" path="/var/lib/kubelet/pods/3d1ea5aa-4aa0-4b2d-a7c6-17def5c877fe/volumes"
Nov 01 04:31:33.600598 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:33.600575    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61039e8c-9bb3-436e-9eff-b19c866b805a" path="/var/lib/kubelet/pods/61039e8c-9bb3-436e-9eff-b19c866b805a/volumes"
Nov 01 04:31:58.731225 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:31:58.731176    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:31:59.961682 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:31:59.961644    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"]
Nov 01 04:32:00.008022 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:00.007990    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-a9c5b3-c6pvx"]
Nov 01 04:32:01.599622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:01.599588    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f03f1b0b-70aa-4d41-a311-8e8c0f8097b6" path="/var/lib/kubelet/pods/f03f1b0b-70aa-4d41-a311-8e8c0f8097b6/volumes"
Nov 01 04:32:02.352690 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:02.352655    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"]
Nov 01 04:32:02.352949 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:02.352921    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy" containerID="cri-o://3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e" gracePeriod=30
Nov 01 04:32:03.204278 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.204241    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"]
Nov 01 04:32:03.204739 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.204725    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ac4811a7-193b-4797-bdcb-7a5f7e22ee63" containerName="revoke-keys"
Nov 01 04:32:03.204797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.204740    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4811a7-193b-4797-bdcb-7a5f7e22ee63" containerName="revoke-keys"
Nov 01 04:32:03.204797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.204758    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a3b071e4-568c-40a8-9a9f-1faa2168aadb" containerName="revoke-keys"
Nov 01 04:32:03.204797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.204763    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b071e4-568c-40a8-9a9f-1faa2168aadb" containerName="revoke-keys"
Nov 01 04:32:03.204912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.204852    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4811a7-193b-4797-bdcb-7a5f7e22ee63" containerName="revoke-keys"
Nov 01 04:32:03.204912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.204863    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b071e4-568c-40a8-9a9f-1faa2168aadb" containerName="revoke-keys"
Nov 01 04:32:03.209346 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.209323    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:03.257466 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.257415    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:32:03.257642 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.257552    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:32:03.306302 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.306265    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"]
Nov 01 04:32:03.309966 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.309938    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdz9\" (UniqueName: \"kubernetes.io/projected/044770b8-77d2-415e-8934-a27f306f847b-kube-api-access-4jdz9\") pod \"maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:03.310076 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.310030    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044770b8-77d2-415e-8934-a27f306f847b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:03.310179 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.310158    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044770b8-77d2-415e-8934-a27f306f847b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:03.411582 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.411544    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044770b8-77d2-415e-8934-a27f306f847b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:03.411783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.411614    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044770b8-77d2-415e-8934-a27f306f847b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:03.411783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.411687    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdz9\" (UniqueName: \"kubernetes.io/projected/044770b8-77d2-415e-8934-a27f306f847b-kube-api-access-4jdz9\") pod \"maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:03.412320 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.412292    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044770b8-77d2-415e-8934-a27f306f847b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:03.414019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.413987    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044770b8-77d2-415e-8934-a27f306f847b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:03.468158 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.468085    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdz9\" (UniqueName: \"kubernetes.io/projected/044770b8-77d2-415e-8934-a27f306f847b-kube-api-access-4jdz9\") pod \"maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:03.522541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.522505    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:03.896564 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:03.896532    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"]
Nov 01 04:32:04.073749 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:04.073712    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8" event={"ID":"044770b8-77d2-415e-8934-a27f306f847b","Type":"ContainerStarted","Data":"4eaa5e9cea6ba21717e6ca37438df3f4e8bbb333ce459f3d11a33904c55ff2b2"}
Nov 01 04:32:05.081717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:05.081620    4882 generic.go:356] "Generic (PLEG): container finished" podID="044770b8-77d2-415e-8934-a27f306f847b" containerID="82167e7db6717a745c6b1523860601bb49dc12f1951801f6e932c045833aa9d4" exitCode=0
Nov 01 04:32:05.081717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:05.081675    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8" event={"ID":"044770b8-77d2-415e-8934-a27f306f847b","Type":"ContainerDied","Data":"82167e7db6717a745c6b1523860601bb49dc12f1951801f6e932c045833aa9d4"}
Nov 01 04:32:06.236055 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:06.236029    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:06.343767 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:06.343682    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/044770b8-77d2-415e-8934-a27f306f847b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044770b8-77d2-415e-8934-a27f306f847b-maas-api-xks-ca\") pod \"044770b8-77d2-415e-8934-a27f306f847b\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") "
Nov 01 04:32:06.343949 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:06.343771    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044770b8-77d2-415e-8934-a27f306f847b-kube-api-access-4jdz9\" (UniqueName: \"kubernetes.io/projected/044770b8-77d2-415e-8934-a27f306f847b-kube-api-access-4jdz9\") pod \"044770b8-77d2-415e-8934-a27f306f847b\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") "
Nov 01 04:32:06.343949 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:06.343839    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044770b8-77d2-415e-8934-a27f306f847b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044770b8-77d2-415e-8934-a27f306f847b-maas-api-openshift-ca\") pod \"044770b8-77d2-415e-8934-a27f306f847b\" (UID: \"044770b8-77d2-415e-8934-a27f306f847b\") "
Nov 01 04:32:06.344267 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:06.344236    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044770b8-77d2-415e-8934-a27f306f847b-maas-api-openshift-ca" pod "044770b8-77d2-415e-8934-a27f306f847b" (UID: "044770b8-77d2-415e-8934-a27f306f847b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:32:06.346084 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:06.346053    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044770b8-77d2-415e-8934-a27f306f847b-maas-api-xks-ca" pod "044770b8-77d2-415e-8934-a27f306f847b" (UID: "044770b8-77d2-415e-8934-a27f306f847b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:32:06.346203 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:06.346175    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044770b8-77d2-415e-8934-a27f306f847b-kube-api-access-4jdz9" pod "044770b8-77d2-415e-8934-a27f306f847b" (UID: "044770b8-77d2-415e-8934-a27f306f847b"). InnerVolumeSpecName "kube-api-access-4jdz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:32:06.445984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:06.445945    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdz9\" (UniqueName: \"kubernetes.io/projected/044770b8-77d2-415e-8934-a27f306f847b-kube-api-access-4jdz9\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:06.445984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:06.445979    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044770b8-77d2-415e-8934-a27f306f847b-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:06.445984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:06.445989    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044770b8-77d2-415e-8934-a27f306f847b-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:07.095096 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.095066    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"
Nov 01 04:32:07.095281 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.095095    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8" event={"ID":"044770b8-77d2-415e-8934-a27f306f847b","Type":"ContainerDied","Data":"4eaa5e9cea6ba21717e6ca37438df3f4e8bbb333ce459f3d11a33904c55ff2b2"}
Nov 01 04:32:07.095281 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.095143    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eaa5e9cea6ba21717e6ca37438df3f4e8bbb333ce459f3d11a33904c55ff2b2"
Nov 01 04:32:07.408740 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.408699    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 04:32:07.409124 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.408759    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 04:32:07.617611 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.617538    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:32:07.760513 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.760476    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-credential-socket\") pod \"d8ca8add-0434-46a9-a055-0306697ac2d7\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") "
Nov 01 04:32:07.760709 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.760578    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-data\") pod \"d8ca8add-0434-46a9-a055-0306697ac2d7\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") "
Nov 01 04:32:07.760709 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.760660    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-envoy\") pod \"d8ca8add-0434-46a9-a055-0306697ac2d7\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") "
Nov 01 04:32:07.760709 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.760695    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-socket\") pod \"d8ca8add-0434-46a9-a055-0306697ac2d7\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") "
Nov 01 04:32:07.760909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.760732    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-kube-api-access-znb8x\" (UniqueName: \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-kube-api-access-znb8x\") pod \"d8ca8add-0434-46a9-a055-0306697ac2d7\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") "
Nov 01 04:32:07.760909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.760758    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-certs\") pod \"d8ca8add-0434-46a9-a055-0306697ac2d7\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") "
Nov 01 04:32:07.760909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.760791    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ca8add-0434-46a9-a055-0306697ac2d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ca8add-0434-46a9-a055-0306697ac2d7-istiod-ca-cert\") pod \"d8ca8add-0434-46a9-a055-0306697ac2d7\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") "
Nov 01 04:32:07.760909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.760798    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-credential-socket" pod "d8ca8add-0434-46a9-a055-0306697ac2d7" (UID: "d8ca8add-0434-46a9-a055-0306697ac2d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:32:07.760909 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.760888    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-data" pod "d8ca8add-0434-46a9-a055-0306697ac2d7" (UID: "d8ca8add-0434-46a9-a055-0306697ac2d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:32:07.761208 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.761094    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-podinfo\") pod \"d8ca8add-0434-46a9-a055-0306697ac2d7\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") "
Nov 01 04:32:07.761208 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.761107    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-socket" pod "d8ca8add-0434-46a9-a055-0306697ac2d7" (UID: "d8ca8add-0434-46a9-a055-0306697ac2d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:32:07.761208 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.761168    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-token\" (UniqueName: \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-token\") pod \"d8ca8add-0434-46a9-a055-0306697ac2d7\" (UID: \"d8ca8add-0434-46a9-a055-0306697ac2d7\") "
Nov 01 04:32:07.761361 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.761211    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-certs" pod "d8ca8add-0434-46a9-a055-0306697ac2d7" (UID: "d8ca8add-0434-46a9-a055-0306697ac2d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:32:07.761361 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.761220    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ca8add-0434-46a9-a055-0306697ac2d7-istiod-ca-cert" pod "d8ca8add-0434-46a9-a055-0306697ac2d7" (UID: "d8ca8add-0434-46a9-a055-0306697ac2d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:32:07.761682 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.761651    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:07.761682 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.761675    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:07.761872 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.761689    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:07.761872 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.761702    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ca8add-0434-46a9-a055-0306697ac2d7-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:07.761872 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.761714    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:07.763545 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.763520    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-envoy" pod "d8ca8add-0434-46a9-a055-0306697ac2d7" (UID: "d8ca8add-0434-46a9-a055-0306697ac2d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:32:07.763751 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.763725    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-podinfo" pod "d8ca8add-0434-46a9-a055-0306697ac2d7" (UID: "d8ca8add-0434-46a9-a055-0306697ac2d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:32:07.763847 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.763769    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-token" pod "d8ca8add-0434-46a9-a055-0306697ac2d7" (UID: "d8ca8add-0434-46a9-a055-0306697ac2d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:32:07.764540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.764518    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-kube-api-access-znb8x" pod "d8ca8add-0434-46a9-a055-0306697ac2d7" (UID: "d8ca8add-0434-46a9-a055-0306697ac2d7"). InnerVolumeSpecName "kube-api-access-znb8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:32:07.862279 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.862241    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:07.862279 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.862272    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:07.862279 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.862284    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ca8add-0434-46a9-a055-0306697ac2d7-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:07.862571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:07.862294    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znb8x\" (UniqueName: \"kubernetes.io/projected/d8ca8add-0434-46a9-a055-0306697ac2d7-kube-api-access-znb8x\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:08.102572 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.102537    4882 generic.go:356] "Generic (PLEG): container finished" podID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerID="3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e" exitCode=0
Nov 01 04:32:08.102754 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.102593    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"
Nov 01 04:32:08.102754 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.102597    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" event={"ID":"d8ca8add-0434-46a9-a055-0306697ac2d7","Type":"ContainerDied","Data":"3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e"}
Nov 01 04:32:08.102754 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.102733    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz" event={"ID":"d8ca8add-0434-46a9-a055-0306697ac2d7","Type":"ContainerDied","Data":"80361abc6f4da6a4a10e2271d267c46621cac99d38315016c15ef5338bb7368a"}
Nov 01 04:32:08.102868 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.102768    4882 scope.go:122] "RemoveContainer" containerID="3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e"
Nov 01 04:32:08.113970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.113947    4882 scope.go:122] "RemoveContainer" containerID="a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714"
Nov 01 04:32:08.126622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.126598    4882 scope.go:122] "RemoveContainer" containerID="3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e"
Nov 01 04:32:08.126988 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:32:08.126967    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e\": container with ID starting with 3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e not found: ID does not exist" containerID="3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e"
Nov 01 04:32:08.127058 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.126998    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e"} err="failed to get container status \"3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e\": rpc error: code = NotFound desc = could not find container \"3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e\": container with ID starting with 3fe9e4b74d16232fad1bbe6c14dbae1edf9f2dc8ef243a0cee882d2d4b02672e not found: ID does not exist"
Nov 01 04:32:08.127058 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.127019    4882 scope.go:122] "RemoveContainer" containerID="a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714"
Nov 01 04:32:08.127284 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:32:08.127265    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714\": container with ID starting with a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714 not found: ID does not exist" containerID="a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714"
Nov 01 04:32:08.127338 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.127287    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714"} err="failed to get container status \"a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714\": rpc error: code = NotFound desc = could not find container \"a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714\": container with ID starting with a90fc79a0e5de995849c25cebca55bc1ad101b87ce6bbc23f6d8d552f3fd6714 not found: ID does not exist"
Nov 01 04:32:08.256493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.256458    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"]
Nov 01 04:32:08.261591 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:08.261565    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ad9e22-openshift-default-6dfdc6589-bsphz"]
Nov 01 04:32:09.599269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:09.599234    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" path="/var/lib/kubelet/pods/d8ca8add-0434-46a9-a055-0306697ac2d7/volumes"
Nov 01 04:32:13.121486 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:13.121450    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:32:13.131571 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:13.131535    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-j945n_b5c8ccd8-df33-4df9-8311-5a163e8e31b3/console-operator/1.log"
Nov 01 04:32:13.914649 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:13.914616    4882 scope.go:122] "RemoveContainer" containerID="1587910ac3e3a835a0ab016dccda508671cb847cd5489d5e1165afbb227133f0"
Nov 01 04:32:13.924342 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:13.924312    4882 scope.go:122] "RemoveContainer" containerID="a18cd009376ef9ce507bf6a3be1523f34fca950ae2bfcf03ac6119bb5215bdb5"
Nov 01 04:32:14.657731 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:14.657695    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"]
Nov 01 04:32:19.313815 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:19.313779    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"]
Nov 01 04:32:19.359501 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:19.359467    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f7545f-f5lv8"]
Nov 01 04:32:19.599632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:19.599562    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044770b8-77d2-415e-8934-a27f306f847b" path="/var/lib/kubelet/pods/044770b8-77d2-415e-8934-a27f306f847b/volumes"
Nov 01 04:32:19.659042 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:19.659004    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz" podUID="c5f6e706-64f9-4e0d-87bd-50f3061fe651" containerName="payload-processing" containerID="cri-o://451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314" gracePeriod=25
Nov 01 04:32:19.913839 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:19.913812    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:32:20.085981 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.085949    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f6e706-64f9-4e0d-87bd-50f3061fe651-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f6e706-64f9-4e0d-87bd-50f3061fe651-plugins-config-volume\") pod \"c5f6e706-64f9-4e0d-87bd-50f3061fe651\" (UID: \"c5f6e706-64f9-4e0d-87bd-50f3061fe651\") "
Nov 01 04:32:20.086174 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.086094    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f6e706-64f9-4e0d-87bd-50f3061fe651-kube-api-access-c744c\" (UniqueName: \"kubernetes.io/projected/c5f6e706-64f9-4e0d-87bd-50f3061fe651-kube-api-access-c744c\") pod \"c5f6e706-64f9-4e0d-87bd-50f3061fe651\" (UID: \"c5f6e706-64f9-4e0d-87bd-50f3061fe651\") "
Nov 01 04:32:20.086388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.086358    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f6e706-64f9-4e0d-87bd-50f3061fe651-plugins-config-volume" pod "c5f6e706-64f9-4e0d-87bd-50f3061fe651" (UID: "c5f6e706-64f9-4e0d-87bd-50f3061fe651"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:32:20.088189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.088166    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f6e706-64f9-4e0d-87bd-50f3061fe651-kube-api-access-c744c" pod "c5f6e706-64f9-4e0d-87bd-50f3061fe651" (UID: "c5f6e706-64f9-4e0d-87bd-50f3061fe651"). InnerVolumeSpecName "kube-api-access-c744c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:32:20.176121 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.176042    4882 generic.go:356] "Generic (PLEG): container finished" podID="c5f6e706-64f9-4e0d-87bd-50f3061fe651" containerID="451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314" exitCode=0
Nov 01 04:32:20.176121 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.176095    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"
Nov 01 04:32:20.176308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.176116    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz" event={"ID":"c5f6e706-64f9-4e0d-87bd-50f3061fe651","Type":"ContainerDied","Data":"451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314"}
Nov 01 04:32:20.176308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.176177    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz" event={"ID":"c5f6e706-64f9-4e0d-87bd-50f3061fe651","Type":"ContainerDied","Data":"8f27a87382ec5cad8198a76f257687b2cf8047640c253eb83b0a902ca5d21455"}
Nov 01 04:32:20.176308 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.176200    4882 scope.go:122] "RemoveContainer" containerID="451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314"
Nov 01 04:32:20.187306 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.187278    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c744c\" (UniqueName: \"kubernetes.io/projected/c5f6e706-64f9-4e0d-87bd-50f3061fe651-kube-api-access-c744c\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:20.187468 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.187309    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f6e706-64f9-4e0d-87bd-50f3061fe651-plugins-config-volume\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:20.188739 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.188724    4882 scope.go:122] "RemoveContainer" containerID="451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314"
Nov 01 04:32:20.189008 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:32:20.188986    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314\": container with ID starting with 451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314 not found: ID does not exist" containerID="451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314"
Nov 01 04:32:20.189086 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.189017    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314"} err="failed to get container status \"451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314\": rpc error: code = NotFound desc = could not find container \"451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314\": container with ID starting with 451f519547952ef0d634b45b9e6c51015d4b7f223276d08f7205f13a8dfe3314 not found: ID does not exist"
Nov 01 04:32:20.338676 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.338600    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"]
Nov 01 04:32:20.343309 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:20.343283    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f7545f-88c6646c5-kf6pz"]
Nov 01 04:32:21.599226 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:21.599191    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f6e706-64f9-4e0d-87bd-50f3061fe651" path="/var/lib/kubelet/pods/c5f6e706-64f9-4e0d-87bd-50f3061fe651/volumes"
Nov 01 04:32:26.066124 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.066094    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"]
Nov 01 04:32:26.066549 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.066291    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy" containerID="cri-o://c6fe7821f166e935eeaae51f7c354909cb86e989306b55e04decde8b18253228" gracePeriod=30
Nov 01 04:32:26.493892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.493856    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:32:26.494064 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.493920    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:32:26.943674 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.943641    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"]
Nov 01 04:32:26.944107 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.944095    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c5f6e706-64f9-4e0d-87bd-50f3061fe651" containerName="payload-processing"
Nov 01 04:32:26.944153 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.944109    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f6e706-64f9-4e0d-87bd-50f3061fe651" containerName="payload-processing"
Nov 01 04:32:26.944153 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.944121    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy"
Nov 01 04:32:26.944153 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.944126    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy"
Nov 01 04:32:26.944153 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.944144    4882 cpu_manager.go:414] "containerMap: removing container" podUID="044770b8-77d2-415e-8934-a27f306f847b" containerName="revoke-keys"
Nov 01 04:32:26.944153 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.944149    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044770b8-77d2-415e-8934-a27f306f847b" containerName="revoke-keys"
Nov 01 04:32:26.944315 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.944236    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="044770b8-77d2-415e-8934-a27f306f847b" containerName="revoke-keys"
Nov 01 04:32:26.944315 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.944244    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy"
Nov 01 04:32:26.944315 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.944256    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f6e706-64f9-4e0d-87bd-50f3061fe651" containerName="payload-processing"
Nov 01 04:32:26.944315 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.944262    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy"
Nov 01 04:32:26.948960 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.948940    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:26.997190 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.997170    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:32:26.997496 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:26.997475    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:32:27.004826 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.004805    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"]
Nov 01 04:32:27.054195 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.054163    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea533d-8101-4936-9241-605c6236aa97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:27.054332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.054200    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea533d-8101-4936-9241-605c6236aa97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:27.054390 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.054373    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzkhf\" (UniqueName: \"kubernetes.io/projected/8fea533d-8101-4936-9241-605c6236aa97-kube-api-access-kzkhf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:27.155890 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.155853    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea533d-8101-4936-9241-605c6236aa97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:27.156260 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.155896    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea533d-8101-4936-9241-605c6236aa97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:27.156260 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.155949    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzkhf\" (UniqueName: \"kubernetes.io/projected/8fea533d-8101-4936-9241-605c6236aa97-kube-api-access-kzkhf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:27.156604 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.156572    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea533d-8101-4936-9241-605c6236aa97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:27.158263 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.158239    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea533d-8101-4936-9241-605c6236aa97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:27.212812 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.212755    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzkhf\" (UniqueName: \"kubernetes.io/projected/8fea533d-8101-4936-9241-605c6236aa97-kube-api-access-kzkhf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:27.262411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.262382    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:27.632152 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:27.632124    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"]
Nov 01 04:32:27.634886 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:32:27.634843    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fea533d_8101_4936_9241_605c6236aa97.slice/crio-1d88266e5e1d3678749560b67f763ae34358d3af3e84b9f1874c1362a9ed7349 WatchSource:0}: Error finding container 1d88266e5e1d3678749560b67f763ae34358d3af3e84b9f1874c1362a9ed7349: Status 404 returned error can't find the container with id 1d88266e5e1d3678749560b67f763ae34358d3af3e84b9f1874c1362a9ed7349
Nov 01 04:32:28.229118 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:28.229029    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk" event={"ID":"8fea533d-8101-4936-9241-605c6236aa97","Type":"ContainerStarted","Data":"05d16ac7f95545e276e60b6e52b133f5a0529c7e533245bbb855849fc1953901"}
Nov 01 04:32:28.229118 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:28.229091    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk" event={"ID":"8fea533d-8101-4936-9241-605c6236aa97","Type":"ContainerStarted","Data":"1d88266e5e1d3678749560b67f763ae34358d3af3e84b9f1874c1362a9ed7349"}
Nov 01 04:32:28.337632 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:28.337575    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk" podStartSLOduration=2.337560236 podStartE2EDuration="2.337560236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:32:28.336623164 +0000 UTC m=+3016.898656179" watchObservedRunningTime="2026-11-01 04:32:28.337560236 +0000 UTC m=+3016.899593253"
Nov 01 04:32:28.721684 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:32:28.721645    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:32:29.236444 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:29.236402    4882 generic.go:356] "Generic (PLEG): container finished" podID="8fea533d-8101-4936-9241-605c6236aa97" containerID="05d16ac7f95545e276e60b6e52b133f5a0529c7e533245bbb855849fc1953901" exitCode=0
Nov 01 04:32:29.236824 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:29.236487    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk" event={"ID":"8fea533d-8101-4936-9241-605c6236aa97","Type":"ContainerDied","Data":"05d16ac7f95545e276e60b6e52b133f5a0529c7e533245bbb855849fc1953901"}
Nov 01 04:32:30.384642 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:30.384619    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:30.387764 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:30.387743    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fea533d-8101-4936-9241-605c6236aa97-kube-api-access-kzkhf\" (UniqueName: \"kubernetes.io/projected/8fea533d-8101-4936-9241-605c6236aa97-kube-api-access-kzkhf\") pod \"8fea533d-8101-4936-9241-605c6236aa97\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") "
Nov 01 04:32:30.389759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:30.389731    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fea533d-8101-4936-9241-605c6236aa97-kube-api-access-kzkhf" pod "8fea533d-8101-4936-9241-605c6236aa97" (UID: "8fea533d-8101-4936-9241-605c6236aa97"). InnerVolumeSpecName "kube-api-access-kzkhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:32:30.488808 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:30.488773    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fea533d-8101-4936-9241-605c6236aa97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea533d-8101-4936-9241-605c6236aa97-maas-api-openshift-ca\") pod \"8fea533d-8101-4936-9241-605c6236aa97\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") "
Nov 01 04:32:30.488998 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:30.488818    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fea533d-8101-4936-9241-605c6236aa97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea533d-8101-4936-9241-605c6236aa97-maas-api-xks-ca\") pod \"8fea533d-8101-4936-9241-605c6236aa97\" (UID: \"8fea533d-8101-4936-9241-605c6236aa97\") "
Nov 01 04:32:30.489081 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:30.489033    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzkhf\" (UniqueName: \"kubernetes.io/projected/8fea533d-8101-4936-9241-605c6236aa97-kube-api-access-kzkhf\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:30.489142 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:30.489106    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fea533d-8101-4936-9241-605c6236aa97-maas-api-openshift-ca" pod "8fea533d-8101-4936-9241-605c6236aa97" (UID: "8fea533d-8101-4936-9241-605c6236aa97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:32:30.490858 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:30.490841    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fea533d-8101-4936-9241-605c6236aa97-maas-api-xks-ca" pod "8fea533d-8101-4936-9241-605c6236aa97" (UID: "8fea533d-8101-4936-9241-605c6236aa97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:32:30.589506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:30.589474    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea533d-8101-4936-9241-605c6236aa97-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:30.589506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:30.589503    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea533d-8101-4936-9241-605c6236aa97-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:31.252113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.252086    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"
Nov 01 04:32:31.252275 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.252081    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk" event={"ID":"8fea533d-8101-4936-9241-605c6236aa97","Type":"ContainerDied","Data":"1d88266e5e1d3678749560b67f763ae34358d3af3e84b9f1874c1362a9ed7349"}
Nov 01 04:32:31.252275 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.252221    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d88266e5e1d3678749560b67f763ae34358d3af3e84b9f1874c1362a9ed7349"
Nov 01 04:32:31.254682 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.254656    4882 generic.go:356] "Generic (PLEG): container finished" podID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerID="c6fe7821f166e935eeaae51f7c354909cb86e989306b55e04decde8b18253228" exitCode=0
Nov 01 04:32:31.254808 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.254724    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" event={"ID":"9df6b1cd-265d-417d-b4c5-13cb3247a318","Type":"ContainerDied","Data":"c6fe7821f166e935eeaae51f7c354909cb86e989306b55e04decde8b18253228"}
Nov 01 04:32:31.254808 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.254778    4882 scope.go:122] "RemoveContainer" containerID="626db13f594b62a08c7b1ec8b52ff1edbdf55998de0fbc59234c2991199a3079"
Nov 01 04:32:31.308260 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.308233    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:32:31.396149 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396123    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-envoy\") pod \"9df6b1cd-265d-417d-b4c5-13cb3247a318\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") "
Nov 01 04:32:31.396583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396173    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-data\") pod \"9df6b1cd-265d-417d-b4c5-13cb3247a318\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") "
Nov 01 04:32:31.396583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396202    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-token\" (UniqueName: \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-token\") pod \"9df6b1cd-265d-417d-b4c5-13cb3247a318\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") "
Nov 01 04:32:31.396583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396264    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-podinfo\") pod \"9df6b1cd-265d-417d-b4c5-13cb3247a318\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") "
Nov 01 04:32:31.396583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396306    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-certs\") pod \"9df6b1cd-265d-417d-b4c5-13cb3247a318\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") "
Nov 01 04:32:31.396583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396338    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-socket\") pod \"9df6b1cd-265d-417d-b4c5-13cb3247a318\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") "
Nov 01 04:32:31.396583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396377    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-credential-socket\") pod \"9df6b1cd-265d-417d-b4c5-13cb3247a318\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") "
Nov 01 04:32:31.396583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396453    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-kube-api-access-bb2rj\" (UniqueName: \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-kube-api-access-bb2rj\") pod \"9df6b1cd-265d-417d-b4c5-13cb3247a318\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") "
Nov 01 04:32:31.396583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396475    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-data" pod "9df6b1cd-265d-417d-b4c5-13cb3247a318" (UID: "9df6b1cd-265d-417d-b4c5-13cb3247a318"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:32:31.396583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396533    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df6b1cd-265d-417d-b4c5-13cb3247a318-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df6b1cd-265d-417d-b4c5-13cb3247a318-istiod-ca-cert\") pod \"9df6b1cd-265d-417d-b4c5-13cb3247a318\" (UID: \"9df6b1cd-265d-417d-b4c5-13cb3247a318\") "
Nov 01 04:32:31.397088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396657    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-certs" pod "9df6b1cd-265d-417d-b4c5-13cb3247a318" (UID: "9df6b1cd-265d-417d-b4c5-13cb3247a318"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:32:31.397088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396673    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-socket" pod "9df6b1cd-265d-417d-b4c5-13cb3247a318" (UID: "9df6b1cd-265d-417d-b4c5-13cb3247a318"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:32:31.397088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396686    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-credential-socket" pod "9df6b1cd-265d-417d-b4c5-13cb3247a318" (UID: "9df6b1cd-265d-417d-b4c5-13cb3247a318"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:32:31.397088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396986    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:31.397088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.397019    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:31.397088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.397035    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:31.397088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.397048    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:31.397088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.396986    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df6b1cd-265d-417d-b4c5-13cb3247a318-istiod-ca-cert" pod "9df6b1cd-265d-417d-b4c5-13cb3247a318" (UID: "9df6b1cd-265d-417d-b4c5-13cb3247a318"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:32:31.398583 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.398556    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-podinfo" pod "9df6b1cd-265d-417d-b4c5-13cb3247a318" (UID: "9df6b1cd-265d-417d-b4c5-13cb3247a318"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:32:31.398679 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.398614    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-envoy" pod "9df6b1cd-265d-417d-b4c5-13cb3247a318" (UID: "9df6b1cd-265d-417d-b4c5-13cb3247a318"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:32:31.398679 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.398665    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-token" pod "9df6b1cd-265d-417d-b4c5-13cb3247a318" (UID: "9df6b1cd-265d-417d-b4c5-13cb3247a318"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:32:31.398758 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.398687    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-kube-api-access-bb2rj" pod "9df6b1cd-265d-417d-b4c5-13cb3247a318" (UID: "9df6b1cd-265d-417d-b4c5-13cb3247a318"). InnerVolumeSpecName "kube-api-access-bb2rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:32:31.497637 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.497607    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:31.497637 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.497632    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb2rj\" (UniqueName: \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-kube-api-access-bb2rj\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:31.497637 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.497642    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df6b1cd-265d-417d-b4c5-13cb3247a318-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:31.497891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.497650    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:31.497891 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:31.497660    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df6b1cd-265d-417d-b4c5-13cb3247a318-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:32:32.262949 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:32.262917    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"
Nov 01 04:32:32.263140 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:32.262952    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl" event={"ID":"9df6b1cd-265d-417d-b4c5-13cb3247a318","Type":"ContainerDied","Data":"6c8e1c229761e3a02b09bce02739f7817ed59eb157c2d2a84c78d91bb4a7c746"}
Nov 01 04:32:32.263140 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:32.263020    4882 scope.go:122] "RemoveContainer" containerID="c6fe7821f166e935eeaae51f7c354909cb86e989306b55e04decde8b18253228"
Nov 01 04:32:32.462875 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:32.462844    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"]
Nov 01 04:32:32.508872 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:32.508839    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f7545f-openshift-default-6846cb4869-p76tl"]
Nov 01 04:32:33.600163 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:33.600132    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" path="/var/lib/kubelet/pods/9df6b1cd-265d-417d-b4c5-13cb3247a318/volumes"
Nov 01 04:32:53.732519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:53.732484    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"]
Nov 01 04:32:53.778870 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:53.778837    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4adf4c-95vqk"]
Nov 01 04:32:55.599074 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:32:55.599041    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fea533d-8101-4936-9241-605c6236aa97" path="/var/lib/kubelet/pods/8fea533d-8101-4936-9241-605c6236aa97/volumes"
Nov 01 04:32:58.724749 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:32:58.724709    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:33:00.525003 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:00.524966    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"]
Nov 01 04:33:00.525519 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:00.525227    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy" containerID="cri-o://64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc" gracePeriod=30
Nov 01 04:33:01.402624 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.402590    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"]
Nov 01 04:33:01.403092 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.403073    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8fea533d-8101-4936-9241-605c6236aa97" containerName="revoke-keys"
Nov 01 04:33:01.403205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.403094    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fea533d-8101-4936-9241-605c6236aa97" containerName="revoke-keys"
Nov 01 04:33:01.403205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.403123    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy"
Nov 01 04:33:01.403205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.403131    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy"
Nov 01 04:33:01.403205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.403164    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy"
Nov 01 04:33:01.403205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.403172    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ca8add-0434-46a9-a055-0306697ac2d7" containerName="istio-proxy"
Nov 01 04:33:01.403205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.403189    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy"
Nov 01 04:33:01.403205 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.403195    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy"
Nov 01 04:33:01.403597 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.403320    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy"
Nov 01 04:33:01.403597 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.403338    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fea533d-8101-4936-9241-605c6236aa97" containerName="revoke-keys"
Nov 01 04:33:01.407883 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.407863    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:01.456050 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.456028    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:33:01.499376 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.499330    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:01.499576 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.499393    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:01.499576 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.499413    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t94b\" (UniqueName: \"kubernetes.io/projected/394eacfa-a405-48d2-be29-21f2f97c62f3-kube-api-access-4t94b\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:01.502115 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.502095    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:33:01.504783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.504764    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"]
Nov 01 04:33:01.600914 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.600859    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:01.600914 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.600914    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t94b\" (UniqueName: \"kubernetes.io/projected/394eacfa-a405-48d2-be29-21f2f97c62f3-kube-api-access-4t94b\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:01.601398 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.601065    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:01.601716 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.601695    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:01.603161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.603139    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:01.657791 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.657723    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t94b\" (UniqueName: \"kubernetes.io/projected/394eacfa-a405-48d2-be29-21f2f97c62f3-kube-api-access-4t94b\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:01.721825 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:01.721787    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:02.100288 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:02.100262    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"]
Nov 01 04:33:02.103177 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:33:02.103145    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394eacfa_a405_48d2_be29_21f2f97c62f3.slice/crio-b0006aa959bc23a9b2af561a8b6f0b4fa6ec1c2bc264990f22a7aebb68e313a8 WatchSource:0}: Error finding container b0006aa959bc23a9b2af561a8b6f0b4fa6ec1c2bc264990f22a7aebb68e313a8: Status 404 returned error can't find the container with id b0006aa959bc23a9b2af561a8b6f0b4fa6ec1c2bc264990f22a7aebb68e313a8
Nov 01 04:33:02.117411 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:02.117385    4882 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:33:02.117531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:02.117443    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:33:02.437347 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:02.437306    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp" event={"ID":"394eacfa-a405-48d2-be29-21f2f97c62f3","Type":"ContainerStarted","Data":"9ba4d452577afe4d88e595641c33b98ecc4874bb0a32eb1577915fb9002c1c94"}
Nov 01 04:33:02.437550 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:02.437412    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp" event={"ID":"394eacfa-a405-48d2-be29-21f2f97c62f3","Type":"ContainerStarted","Data":"b0006aa959bc23a9b2af561a8b6f0b4fa6ec1c2bc264990f22a7aebb68e313a8"}
Nov 01 04:33:02.546586 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:02.546535    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp" podStartSLOduration=1.546521259 podStartE2EDuration="1.546521259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:33:02.544456965 +0000 UTC m=+3051.106489979" watchObservedRunningTime="2026-11-01 04:33:02.546521259 +0000 UTC m=+3051.108554275"
Nov 01 04:33:03.444517 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:03.444413    4882 generic.go:356] "Generic (PLEG): container finished" podID="394eacfa-a405-48d2-be29-21f2f97c62f3" containerID="9ba4d452577afe4d88e595641c33b98ecc4874bb0a32eb1577915fb9002c1c94" exitCode=0
Nov 01 04:33:03.444517 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:03.444461    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp" event={"ID":"394eacfa-a405-48d2-be29-21f2f97c62f3","Type":"ContainerDied","Data":"9ba4d452577afe4d88e595641c33b98ecc4874bb0a32eb1577915fb9002c1c94"}
Nov 01 04:33:04.589153 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:04.589129    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:04.734113 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:04.734030    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-xks-ca\") pod \"394eacfa-a405-48d2-be29-21f2f97c62f3\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") "
Nov 01 04:33:04.734284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:04.734125    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394eacfa-a405-48d2-be29-21f2f97c62f3-kube-api-access-4t94b\" (UniqueName: \"kubernetes.io/projected/394eacfa-a405-48d2-be29-21f2f97c62f3-kube-api-access-4t94b\") pod \"394eacfa-a405-48d2-be29-21f2f97c62f3\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") "
Nov 01 04:33:04.734284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:04.734199    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-openshift-ca\") pod \"394eacfa-a405-48d2-be29-21f2f97c62f3\" (UID: \"394eacfa-a405-48d2-be29-21f2f97c62f3\") "
Nov 01 04:33:04.734666 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:04.734621    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-openshift-ca" pod "394eacfa-a405-48d2-be29-21f2f97c62f3" (UID: "394eacfa-a405-48d2-be29-21f2f97c62f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:33:04.736401 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:04.736382    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394eacfa-a405-48d2-be29-21f2f97c62f3-kube-api-access-4t94b" pod "394eacfa-a405-48d2-be29-21f2f97c62f3" (UID: "394eacfa-a405-48d2-be29-21f2f97c62f3"). InnerVolumeSpecName "kube-api-access-4t94b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:33:04.736655 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:04.736631    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-xks-ca" pod "394eacfa-a405-48d2-be29-21f2f97c62f3" (UID: "394eacfa-a405-48d2-be29-21f2f97c62f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:33:04.835537 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:04.835504    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:04.835537 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:04.835532    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t94b\" (UniqueName: \"kubernetes.io/projected/394eacfa-a405-48d2-be29-21f2f97c62f3-kube-api-access-4t94b\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:04.835537 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:04.835543    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394eacfa-a405-48d2-be29-21f2f97c62f3-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:05.457785 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.457757    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"
Nov 01 04:33:05.457978 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.457758    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp" event={"ID":"394eacfa-a405-48d2-be29-21f2f97c62f3","Type":"ContainerDied","Data":"b0006aa959bc23a9b2af561a8b6f0b4fa6ec1c2bc264990f22a7aebb68e313a8"}
Nov 01 04:33:05.457978 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.457887    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0006aa959bc23a9b2af561a8b6f0b4fa6ec1c2bc264990f22a7aebb68e313a8"
Nov 01 04:33:05.788349 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.788326    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:33:05.843984 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.843959    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-socket\") pod \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") "
Nov 01 04:33:05.844100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844007    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-envoy\") pod \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") "
Nov 01 04:33:05.844100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844032    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-credential-socket\") pod \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") "
Nov 01 04:33:05.844100 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844051    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istiod-ca-cert\") pod \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") "
Nov 01 04:33:05.844269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844100    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-certs\") pod \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") "
Nov 01 04:33:05.844269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844144    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-data\") pod \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") "
Nov 01 04:33:05.844269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844184    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-kube-api-access-fpwwr\" (UniqueName: \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-kube-api-access-fpwwr\") pod \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") "
Nov 01 04:33:05.844269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844203    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-token\" (UniqueName: \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-token\") pod \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") "
Nov 01 04:33:05.844269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844249    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-podinfo\") pod \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\" (UID: \"ad0056df-1fe9-42e7-8e0f-ea47f63a1243\") "
Nov 01 04:33:05.844628 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844244    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-socket" pod "ad0056df-1fe9-42e7-8e0f-ea47f63a1243" (UID: "ad0056df-1fe9-42e7-8e0f-ea47f63a1243"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:05.844628 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844363    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-credential-socket" pod "ad0056df-1fe9-42e7-8e0f-ea47f63a1243" (UID: "ad0056df-1fe9-42e7-8e0f-ea47f63a1243"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:05.844628 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844459    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-certs" pod "ad0056df-1fe9-42e7-8e0f-ea47f63a1243" (UID: "ad0056df-1fe9-42e7-8e0f-ea47f63a1243"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:05.844628 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844508    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istiod-ca-cert" pod "ad0056df-1fe9-42e7-8e0f-ea47f63a1243" (UID: "ad0056df-1fe9-42e7-8e0f-ea47f63a1243"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:33:05.844848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844739    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:05.844848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844763    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:05.844848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844779    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:05.844848 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.844793    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:05.845055 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.845019    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-data" pod "ad0056df-1fe9-42e7-8e0f-ea47f63a1243" (UID: "ad0056df-1fe9-42e7-8e0f-ea47f63a1243"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:05.846326 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.846309    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-podinfo" pod "ad0056df-1fe9-42e7-8e0f-ea47f63a1243" (UID: "ad0056df-1fe9-42e7-8e0f-ea47f63a1243"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:33:05.846455 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.846415    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-token" pod "ad0056df-1fe9-42e7-8e0f-ea47f63a1243" (UID: "ad0056df-1fe9-42e7-8e0f-ea47f63a1243"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:33:05.846648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.846629    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-envoy" pod "ad0056df-1fe9-42e7-8e0f-ea47f63a1243" (UID: "ad0056df-1fe9-42e7-8e0f-ea47f63a1243"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:05.846692 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.846646    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-kube-api-access-fpwwr" pod "ad0056df-1fe9-42e7-8e0f-ea47f63a1243" (UID: "ad0056df-1fe9-42e7-8e0f-ea47f63a1243"). InnerVolumeSpecName "kube-api-access-fpwwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:33:05.946132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.946095    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:05.946132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.946133    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:05.946310 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.946149    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpwwr\" (UniqueName: \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-kube-api-access-fpwwr\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:05.946310 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.946166    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:05.946310 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:05.946180    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0056df-1fe9-42e7-8e0f-ea47f63a1243-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:06.465562 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.465524    4882 generic.go:356] "Generic (PLEG): container finished" podID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerID="64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc" exitCode=0
Nov 01 04:33:06.465737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.465578    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"
Nov 01 04:33:06.465737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.465613    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" event={"ID":"ad0056df-1fe9-42e7-8e0f-ea47f63a1243","Type":"ContainerDied","Data":"64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc"}
Nov 01 04:33:06.465737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.465668    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x" event={"ID":"ad0056df-1fe9-42e7-8e0f-ea47f63a1243","Type":"ContainerDied","Data":"7ee82a1569d1520265f82cbe2d12213c90ff1918e8e4020b25ae62e4386f7f69"}
Nov 01 04:33:06.465737 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.465693    4882 scope.go:122] "RemoveContainer" containerID="64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc"
Nov 01 04:33:06.478708 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.478689    4882 scope.go:122] "RemoveContainer" containerID="44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a"
Nov 01 04:33:06.487525 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.487502    4882 scope.go:122] "RemoveContainer" containerID="64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc"
Nov 01 04:33:06.487805 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:33:06.487786    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc\": container with ID starting with 64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc not found: ID does not exist" containerID="64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc"
Nov 01 04:33:06.487853 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.487816    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc"} err="failed to get container status \"64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc\": rpc error: code = NotFound desc = could not find container \"64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc\": container with ID starting with 64dacabb4d5c65a7a22343f6c75a452517bee905964b5f5cdca20c71e8fedefc not found: ID does not exist"
Nov 01 04:33:06.487853 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.487834    4882 scope.go:122] "RemoveContainer" containerID="44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a"
Nov 01 04:33:06.488065 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:33:06.488044    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a\": container with ID starting with 44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a not found: ID does not exist" containerID="44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a"
Nov 01 04:33:06.488151 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.488069    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a"} err="failed to get container status \"44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a\": rpc error: code = NotFound desc = could not find container \"44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a\": container with ID starting with 44c1a0ebdd87a3e793b9c5e3f5c852b39f2a5b1a3139d9349819b7d7a96d828a not found: ID does not exist"
Nov 01 04:33:06.619483 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.619451    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"]
Nov 01 04:33:06.665648 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:06.665619    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4adf4c-openshift-default-647bb7f855-gtf5x"]
Nov 01 04:33:07.599298 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:07.599266    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" path="/var/lib/kubelet/pods/ad0056df-1fe9-42e7-8e0f-ea47f63a1243/volumes"
Nov 01 04:33:13.955992 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:13.955961    4882 scope.go:122] "RemoveContainer" containerID="e4c6bab24d162ef5525c0243e4bdbec024215a32e35cc87df6257eb7f8e0dc58"
Nov 01 04:33:13.964958 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:13.964935    4882 scope.go:122] "RemoveContainer" containerID="76350034e4b3313c6e4678d1c65a0c63ec94c5cfefd5eadada553e34ce20f2a2"
Nov 01 04:33:13.973384 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:13.973363    4882 scope.go:122] "RemoveContainer" containerID="239513fa895f48efd30d86f25ab34fe5779f1889c61c23cd74d90586076f9a9a"
Nov 01 04:33:13.981649 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:13.981627    4882 scope.go:122] "RemoveContainer" containerID="c0ed03aca37557763492c51bcb4eac515913ec020731f5d6da51e02529e238ac"
Nov 01 04:33:25.545987 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:25.545951    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"]
Nov 01 04:33:25.591734 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:25.591702    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fb1c1f-f6dlp"]
Nov 01 04:33:25.598120 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:25.598094    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394eacfa-a405-48d2-be29-21f2f97c62f3" path="/var/lib/kubelet/pods/394eacfa-a405-48d2-be29-21f2f97c62f3/volumes"
Nov 01 04:33:28.721972 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:33:28.721931    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:33:33.356232 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.356197    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"]
Nov 01 04:33:33.356700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.356660    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy"
Nov 01 04:33:33.356700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.356672    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy"
Nov 01 04:33:33.356700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.356684    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy"
Nov 01 04:33:33.356700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.356689    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy"
Nov 01 04:33:33.356700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.356701    4882 cpu_manager.go:414] "containerMap: removing container" podUID="394eacfa-a405-48d2-be29-21f2f97c62f3" containerName="revoke-keys"
Nov 01 04:33:33.356888 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.356706    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394eacfa-a405-48d2-be29-21f2f97c62f3" containerName="revoke-keys"
Nov 01 04:33:33.356888 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.356784    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy"
Nov 01 04:33:33.356888 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.356791    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0056df-1fe9-42e7-8e0f-ea47f63a1243" containerName="istio-proxy"
Nov 01 04:33:33.356888 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.356800    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="394eacfa-a405-48d2-be29-21f2f97c62f3" containerName="revoke-keys"
Nov 01 04:33:33.356888 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.356809    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df6b1cd-265d-417d-b4c5-13cb3247a318" containerName="istio-proxy"
Nov 01 04:33:33.361735 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.361713    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:33.410062 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.410030    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:33:33.410230 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.410171    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:33:33.458092 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.458057    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"]
Nov 01 04:33:33.510816 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.510781    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sldkv\" (UniqueName: \"kubernetes.io/projected/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-kube-api-access-sldkv\") pod \"maas-api-revoke-keys-e2e-worker-w2-865066-hf67v\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:33.510990 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.510838    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-865066-hf67v\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:33.510990 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.510972    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-865066-hf67v\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:33.612228 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.612147    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sldkv\" (UniqueName: \"kubernetes.io/projected/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-kube-api-access-sldkv\") pod \"maas-api-revoke-keys-e2e-worker-w2-865066-hf67v\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:33.612228 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.612219    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-865066-hf67v\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:33.612416 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.612275    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-865066-hf67v\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:33.612887 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.612863    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-865066-hf67v\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:33.614670 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.614646    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-865066-hf67v\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:33.669019 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.668985    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sldkv\" (UniqueName: \"kubernetes.io/projected/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-kube-api-access-sldkv\") pod \"maas-api-revoke-keys-e2e-worker-w2-865066-hf67v\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:33.674640 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:33.674613    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:34.039540 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:34.039513    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"]
Nov 01 04:33:34.042265 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:33:34.042232    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea5b60b_704b_43d3_ba7b_a7d785e2ae1b.slice/crio-41a4af45bce6431f07fca7e4c73255752ce17041df4487e8a3a9f04aaf4ff503 WatchSource:0}: Error finding container 41a4af45bce6431f07fca7e4c73255752ce17041df4487e8a3a9f04aaf4ff503: Status 404 returned error can't find the container with id 41a4af45bce6431f07fca7e4c73255752ce17041df4487e8a3a9f04aaf4ff503
Nov 01 04:33:34.618447 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:34.618387    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v" event={"ID":"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b","Type":"ContainerStarted","Data":"ff56a760d740c77189503ca4d4fd7b937431d409e2816b67a84c8769e4cfb795"}
Nov 01 04:33:34.618826 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:34.618471    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v" event={"ID":"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b","Type":"ContainerStarted","Data":"41a4af45bce6431f07fca7e4c73255752ce17041df4487e8a3a9f04aaf4ff503"}
Nov 01 04:33:34.727245 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:34.727190    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v" podStartSLOduration=1.7271763180000002 podStartE2EDuration="1.727176318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:33:34.726488158 +0000 UTC m=+3083.288521175" watchObservedRunningTime="2026-11-01 04:33:34.727176318 +0000 UTC m=+3083.289209334"
Nov 01 04:33:35.624915 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:35.624879    4882 generic.go:356] "Generic (PLEG): container finished" podID="3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b" containerID="ff56a760d740c77189503ca4d4fd7b937431d409e2816b67a84c8769e4cfb795" exitCode=0
Nov 01 04:33:35.625356 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:35.624958    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v" event={"ID":"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b","Type":"ContainerDied","Data":"ff56a760d740c77189503ca4d4fd7b937431d409e2816b67a84c8769e4cfb795"}
Nov 01 04:33:36.771521 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:36.771490    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:36.942615 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:36.942587    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-xks-ca\") pod \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") "
Nov 01 04:33:36.942783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:36.942770    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-kube-api-access-sldkv\" (UniqueName: \"kubernetes.io/projected/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-kube-api-access-sldkv\") pod \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") "
Nov 01 04:33:36.942862 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:36.942851    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-openshift-ca\") pod \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\" (UID: \"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b\") "
Nov 01 04:33:36.943243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:36.943219    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-openshift-ca" pod "3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b" (UID: "3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:33:36.944695 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:36.944673    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-xks-ca" pod "3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b" (UID: "3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:33:36.944850 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:36.944827    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-kube-api-access-sldkv" pod "3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b" (UID: "3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b"). InnerVolumeSpecName "kube-api-access-sldkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:33:37.043797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:37.043759    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sldkv\" (UniqueName: \"kubernetes.io/projected/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-kube-api-access-sldkv\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:37.043797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:37.043791    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:37.043797 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:37.043802    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:37.636920 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:37.636891    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"
Nov 01 04:33:37.637082 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:37.636889    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v" event={"ID":"3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b","Type":"ContainerDied","Data":"41a4af45bce6431f07fca7e4c73255752ce17041df4487e8a3a9f04aaf4ff503"}
Nov 01 04:33:37.637082 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:37.637009    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41a4af45bce6431f07fca7e4c73255752ce17041df4487e8a3a9f04aaf4ff503"
Nov 01 04:33:37.779770 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:37.779741    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"]
Nov 01 04:33:37.788347 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:37.788317    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"]
Nov 01 04:33:37.810780 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:37.810743    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"]
Nov 01 04:33:45.479291 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.479253    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"]
Nov 01 04:33:45.479744 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.479717    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b" containerName="revoke-keys"
Nov 01 04:33:45.479744 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.479730    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b" containerName="revoke-keys"
Nov 01 04:33:45.479849 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.479838    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b" containerName="revoke-keys"
Nov 01 04:33:45.483139 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.483118    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:45.529200 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.529161    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrlg\" (UniqueName: \"kubernetes.io/projected/0978d086-0b2f-4a5f-a94e-6a67863a9966-kube-api-access-hrrlg\") pod \"maas-api-revoke-sub-87254a4d-hs89k\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:45.529388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.529222    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87254a4d-hs89k\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:45.529388 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.529366    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87254a4d-hs89k\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:45.531564 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.531540    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:33:45.577469 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.577416    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\""
Nov 01 04:33:45.579993 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.579965    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"]
Nov 01 04:33:45.629912 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.629880    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrlg\" (UniqueName: \"kubernetes.io/projected/0978d086-0b2f-4a5f-a94e-6a67863a9966-kube-api-access-hrrlg\") pod \"maas-api-revoke-sub-87254a4d-hs89k\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:45.630078 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.629958    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87254a4d-hs89k\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:45.630078 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.630026    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87254a4d-hs89k\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:45.630970 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.630933    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87254a4d-hs89k\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:45.632248 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.632229    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87254a4d-hs89k\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:45.687517 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.687481    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrlg\" (UniqueName: \"kubernetes.io/projected/0978d086-0b2f-4a5f-a94e-6a67863a9966-kube-api-access-hrrlg\") pod \"maas-api-revoke-sub-87254a4d-hs89k\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:45.796700 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:45.796607    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:46.167129 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:46.167084    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"]
Nov 01 04:33:46.687056 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:46.687017    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k" event={"ID":"0978d086-0b2f-4a5f-a94e-6a67863a9966","Type":"ContainerStarted","Data":"7d43594fd39020d5469e7810fffb24a062410d982cbb10876ee2267eed3e817f"}
Nov 01 04:33:46.687506 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:46.687086    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k" event={"ID":"0978d086-0b2f-4a5f-a94e-6a67863a9966","Type":"ContainerStarted","Data":"1c74824c28a33a6f44d5fb6cbe462c0d3e305bef5e72b22ec2e378f420cdf423"}
Nov 01 04:33:46.795652 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:46.795559    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k" podStartSLOduration=1.7955455059999998 podStartE2EDuration="1.795545506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:33:46.794334676 +0000 UTC m=+3095.356367694" watchObservedRunningTime="2026-11-01 04:33:46.795545506 +0000 UTC m=+3095.357578523"
Nov 01 04:33:47.694313 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:47.694278    4882 generic.go:356] "Generic (PLEG): container finished" podID="0978d086-0b2f-4a5f-a94e-6a67863a9966" containerID="7d43594fd39020d5469e7810fffb24a062410d982cbb10876ee2267eed3e817f" exitCode=0
Nov 01 04:33:47.694752 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:47.694365    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k" event={"ID":"0978d086-0b2f-4a5f-a94e-6a67863a9966","Type":"ContainerDied","Data":"7d43594fd39020d5469e7810fffb24a062410d982cbb10876ee2267eed3e817f"}
Nov 01 04:33:48.828784 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:48.828757    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:48.854849 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:48.854819    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-xks-ca\") pod \"0978d086-0b2f-4a5f-a94e-6a67863a9966\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") "
Nov 01 04:33:48.855001 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:48.854864    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-openshift-ca\") pod \"0978d086-0b2f-4a5f-a94e-6a67863a9966\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") "
Nov 01 04:33:48.855001 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:48.854904    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0978d086-0b2f-4a5f-a94e-6a67863a9966-kube-api-access-hrrlg\" (UniqueName: \"kubernetes.io/projected/0978d086-0b2f-4a5f-a94e-6a67863a9966-kube-api-access-hrrlg\") pod \"0978d086-0b2f-4a5f-a94e-6a67863a9966\" (UID: \"0978d086-0b2f-4a5f-a94e-6a67863a9966\") "
Nov 01 04:33:48.855282 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:48.855260    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-openshift-ca" pod "0978d086-0b2f-4a5f-a94e-6a67863a9966" (UID: "0978d086-0b2f-4a5f-a94e-6a67863a9966"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:33:48.856962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:48.856937    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0978d086-0b2f-4a5f-a94e-6a67863a9966-kube-api-access-hrrlg" pod "0978d086-0b2f-4a5f-a94e-6a67863a9966" (UID: "0978d086-0b2f-4a5f-a94e-6a67863a9966"). InnerVolumeSpecName "kube-api-access-hrrlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:33:48.857020 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:48.856942    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-xks-ca" pod "0978d086-0b2f-4a5f-a94e-6a67863a9966" (UID: "0978d086-0b2f-4a5f-a94e-6a67863a9966"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:33:48.955964 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:48.955891    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-xks-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:48.955964 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:48.955920    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0978d086-0b2f-4a5f-a94e-6a67863a9966-maas-api-openshift-ca\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:48.955964 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:48.955931    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrrlg\" (UniqueName: \"kubernetes.io/projected/0978d086-0b2f-4a5f-a94e-6a67863a9966-kube-api-access-hrrlg\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:49.707243 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:49.707214    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k"
Nov 01 04:33:49.707423 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:49.707259    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87254a4d-hs89k" event={"ID":"0978d086-0b2f-4a5f-a94e-6a67863a9966","Type":"ContainerDied","Data":"1c74824c28a33a6f44d5fb6cbe462c0d3e305bef5e72b22ec2e378f420cdf423"}
Nov 01 04:33:49.707423 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:49.707306    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c74824c28a33a6f44d5fb6cbe462c0d3e305bef5e72b22ec2e378f420cdf423"
Nov 01 04:33:52.800704 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:52.800615    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerName="main" containerID="cri-o://932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166" gracePeriod=45
Nov 01 04:33:52.807549 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:52.807512    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main" containerID="cri-o://77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80" gracePeriod=45
Nov 01 04:33:52.829990 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:52.829935    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main" containerID="cri-o://b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196" gracePeriod=45
Nov 01 04:33:52.996103 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:52.995787    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": dial tcp 10.132.0.75:8000: connect: connection refused"
Nov 01 04:33:53.118995 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.118964    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:33:53.122646 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.122617    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 04:33:53.151209 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.151178    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 04:33:53.204392 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204360    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-tmp-dir\") pod \"f8104497-11eb-49d7-b2b5-328dda72bb22\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") "
Nov 01 04:33:53.204602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204413    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-tmp-dir\") pod \"c92a7a3f-7ff0-42fc-846a-cc386e951275\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") "
Nov 01 04:33:53.204602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204490    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-model-cache\") pod \"f8104497-11eb-49d7-b2b5-328dda72bb22\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") "
Nov 01 04:33:53.204602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204536    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1301538d-17a0-4021-aeda-db3130e114b1-kube-api-access-jjtgj\" (UniqueName: \"kubernetes.io/projected/1301538d-17a0-4021-aeda-db3130e114b1-kube-api-access-jjtgj\") pod \"1301538d-17a0-4021-aeda-db3130e114b1\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") "
Nov 01 04:33:53.204602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204562    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-dshm\") pod \"f8104497-11eb-49d7-b2b5-328dda72bb22\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") "
Nov 01 04:33:53.204602 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204593    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c92a7a3f-7ff0-42fc-846a-cc386e951275-tls-certs\" (UniqueName: \"kubernetes.io/secret/c92a7a3f-7ff0-42fc-846a-cc386e951275-tls-certs\") pod \"c92a7a3f-7ff0-42fc-846a-cc386e951275\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") "
Nov 01 04:33:53.204901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204629    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8104497-11eb-49d7-b2b5-328dda72bb22-kube-api-access-xckcx\" (UniqueName: \"kubernetes.io/projected/f8104497-11eb-49d7-b2b5-328dda72bb22-kube-api-access-xckcx\") pod \"f8104497-11eb-49d7-b2b5-328dda72bb22\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") "
Nov 01 04:33:53.204901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204653    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-tmp-dir" pod "f8104497-11eb-49d7-b2b5-328dda72bb22" (UID: "f8104497-11eb-49d7-b2b5-328dda72bb22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.204901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204662    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1301538d-17a0-4021-aeda-db3130e114b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/1301538d-17a0-4021-aeda-db3130e114b1-tls-certs\") pod \"1301538d-17a0-4021-aeda-db3130e114b1\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") "
Nov 01 04:33:53.204901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204737    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-model-cache\") pod \"c92a7a3f-7ff0-42fc-846a-cc386e951275\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") "
Nov 01 04:33:53.204901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204774    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-tmp-dir" pod "c92a7a3f-7ff0-42fc-846a-cc386e951275" (UID: "c92a7a3f-7ff0-42fc-846a-cc386e951275"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.204901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204788    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92a7a3f-7ff0-42fc-846a-cc386e951275-kube-api-access-pn86v\" (UniqueName: \"kubernetes.io/projected/c92a7a3f-7ff0-42fc-846a-cc386e951275-kube-api-access-pn86v\") pod \"c92a7a3f-7ff0-42fc-846a-cc386e951275\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") "
Nov 01 04:33:53.204901 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204873    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-tmp-dir\") pod \"1301538d-17a0-4021-aeda-db3130e114b1\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") "
Nov 01 04:33:53.205333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204919    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-home\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-home\") pod \"c92a7a3f-7ff0-42fc-846a-cc386e951275\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") "
Nov 01 04:33:53.205333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204957    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-home\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-home\") pod \"1301538d-17a0-4021-aeda-db3130e114b1\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") "
Nov 01 04:33:53.205333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.204984    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-dshm\") pod \"1301538d-17a0-4021-aeda-db3130e114b1\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") "
Nov 01 04:33:53.205333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.205052    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-home\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-home\") pod \"f8104497-11eb-49d7-b2b5-328dda72bb22\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") "
Nov 01 04:33:53.205333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.205119    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-model-cache" pod "c92a7a3f-7ff0-42fc-846a-cc386e951275" (UID: "c92a7a3f-7ff0-42fc-846a-cc386e951275"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.205333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.205135    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8104497-11eb-49d7-b2b5-328dda72bb22-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8104497-11eb-49d7-b2b5-328dda72bb22-tls-certs\") pod \"f8104497-11eb-49d7-b2b5-328dda72bb22\" (UID: \"f8104497-11eb-49d7-b2b5-328dda72bb22\") "
Nov 01 04:33:53.205333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.205172    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-dshm\") pod \"c92a7a3f-7ff0-42fc-846a-cc386e951275\" (UID: \"c92a7a3f-7ff0-42fc-846a-cc386e951275\") "
Nov 01 04:33:53.205333 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.205210    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-model-cache\") pod \"1301538d-17a0-4021-aeda-db3130e114b1\" (UID: \"1301538d-17a0-4021-aeda-db3130e114b1\") "
Nov 01 04:33:53.205795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.205674    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-tmp-dir\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.205795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.205695    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-tmp-dir\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.205795 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.205708    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-model-cache\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.205976 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.205919    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-model-cache" pod "1301538d-17a0-4021-aeda-db3130e114b1" (UID: "1301538d-17a0-4021-aeda-db3130e114b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.206046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.205968    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-model-cache" pod "f8104497-11eb-49d7-b2b5-328dda72bb22" (UID: "f8104497-11eb-49d7-b2b5-328dda72bb22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.206109 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.206052    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-home" pod "c92a7a3f-7ff0-42fc-846a-cc386e951275" (UID: "c92a7a3f-7ff0-42fc-846a-cc386e951275"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.206168 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.206132    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-home" pod "f8104497-11eb-49d7-b2b5-328dda72bb22" (UID: "f8104497-11eb-49d7-b2b5-328dda72bb22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.206256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.206237    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-home" pod "1301538d-17a0-4021-aeda-db3130e114b1" (UID: "1301538d-17a0-4021-aeda-db3130e114b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.206315 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.206277    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-tmp-dir" pod "1301538d-17a0-4021-aeda-db3130e114b1" (UID: "1301538d-17a0-4021-aeda-db3130e114b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.211299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.211267    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8104497-11eb-49d7-b2b5-328dda72bb22-tls-certs" pod "f8104497-11eb-49d7-b2b5-328dda72bb22" (UID: "f8104497-11eb-49d7-b2b5-328dda72bb22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:33:53.211299 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.211277    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1301538d-17a0-4021-aeda-db3130e114b1-kube-api-access-jjtgj" pod "1301538d-17a0-4021-aeda-db3130e114b1" (UID: "1301538d-17a0-4021-aeda-db3130e114b1"). InnerVolumeSpecName "kube-api-access-jjtgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:33:53.211523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.211320    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c92a7a3f-7ff0-42fc-846a-cc386e951275-tls-certs" pod "c92a7a3f-7ff0-42fc-846a-cc386e951275" (UID: "c92a7a3f-7ff0-42fc-846a-cc386e951275"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:33:53.211523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.211361    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1301538d-17a0-4021-aeda-db3130e114b1-tls-certs" pod "1301538d-17a0-4021-aeda-db3130e114b1" (UID: "1301538d-17a0-4021-aeda-db3130e114b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:33:53.211523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.211368    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92a7a3f-7ff0-42fc-846a-cc386e951275-kube-api-access-pn86v" pod "c92a7a3f-7ff0-42fc-846a-cc386e951275" (UID: "c92a7a3f-7ff0-42fc-846a-cc386e951275"). InnerVolumeSpecName "kube-api-access-pn86v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:33:53.211523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.211448    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8104497-11eb-49d7-b2b5-328dda72bb22-kube-api-access-xckcx" pod "f8104497-11eb-49d7-b2b5-328dda72bb22" (UID: "f8104497-11eb-49d7-b2b5-328dda72bb22"). InnerVolumeSpecName "kube-api-access-xckcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:33:53.211523 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.211472    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-dshm" pod "f8104497-11eb-49d7-b2b5-328dda72bb22" (UID: "f8104497-11eb-49d7-b2b5-328dda72bb22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.211701 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.211525    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-dshm" pod "1301538d-17a0-4021-aeda-db3130e114b1" (UID: "1301538d-17a0-4021-aeda-db3130e114b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.212474 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.212449    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-dshm" pod "c92a7a3f-7ff0-42fc-846a-cc386e951275" (UID: "c92a7a3f-7ff0-42fc-846a-cc386e951275"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:33:53.306738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306660    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjtgj\" (UniqueName: \"kubernetes.io/projected/1301538d-17a0-4021-aeda-db3130e114b1-kube-api-access-jjtgj\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.306738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306690    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-dshm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.306738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306700    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c92a7a3f-7ff0-42fc-846a-cc386e951275-tls-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.306738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306709    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xckcx\" (UniqueName: \"kubernetes.io/projected/f8104497-11eb-49d7-b2b5-328dda72bb22-kube-api-access-xckcx\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.306738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306718    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1301538d-17a0-4021-aeda-db3130e114b1-tls-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.306738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306727    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn86v\" (UniqueName: \"kubernetes.io/projected/c92a7a3f-7ff0-42fc-846a-cc386e951275-kube-api-access-pn86v\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.306738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306735    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-tmp-dir\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.306738 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306744    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-home\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.307106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306752    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-home\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.307106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306760    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-dshm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.307106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306767    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-home\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.307106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306776    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8104497-11eb-49d7-b2b5-328dda72bb22-tls-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.307106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306785    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c92a7a3f-7ff0-42fc-846a-cc386e951275-dshm\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.307106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306794    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1301538d-17a0-4021-aeda-db3130e114b1-model-cache\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.307106 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.306803    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8104497-11eb-49d7-b2b5-328dda72bb22-model-cache\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:33:53.731528 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.731491    4882 generic.go:356] "Generic (PLEG): container finished" podID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerID="932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166" exitCode=0
Nov 01 04:33:53.731736 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.731540    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"
Nov 01 04:33:53.731736 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.731574    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" event={"ID":"c92a7a3f-7ff0-42fc-846a-cc386e951275","Type":"ContainerDied","Data":"932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166"}
Nov 01 04:33:53.731736 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.731626    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" event={"ID":"c92a7a3f-7ff0-42fc-846a-cc386e951275","Type":"ContainerDied","Data":"b23559165e663cf167f4c69bfa4ab85bd52d4c7782261d05f00b049f7dbed792"}
Nov 01 04:33:53.731736 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.731650    4882 scope.go:122] "RemoveContainer" containerID="932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166"
Nov 01 04:33:53.732999 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.732974    4882 generic.go:356] "Generic (PLEG): container finished" podID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerID="77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80" exitCode=0
Nov 01 04:33:53.733127 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.733016    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"
Nov 01 04:33:53.733127 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.733054    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" event={"ID":"f8104497-11eb-49d7-b2b5-328dda72bb22","Type":"ContainerDied","Data":"77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80"}
Nov 01 04:33:53.733127 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.733094    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg" event={"ID":"f8104497-11eb-49d7-b2b5-328dda72bb22","Type":"ContainerDied","Data":"b7b2e24e3d43988b92eb31e0afbcb0aaa50018265018447ce64f1f23cad2de73"}
Nov 01 04:33:53.734719 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.734696    4882 generic.go:356] "Generic (PLEG): container finished" podID="1301538d-17a0-4021-aeda-db3130e114b1" containerID="b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196" exitCode=0
Nov 01 04:33:53.734844 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.734724    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"
Nov 01 04:33:53.734844 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.734775    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" event={"ID":"1301538d-17a0-4021-aeda-db3130e114b1","Type":"ContainerDied","Data":"b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196"}
Nov 01 04:33:53.734844 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.734813    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc" event={"ID":"1301538d-17a0-4021-aeda-db3130e114b1","Type":"ContainerDied","Data":"7329c7ef8511329cc16826feb6e8499fcb0f1ed5cb8233a58ef7bba6a16f3d80"}
Nov 01 04:33:53.745531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.745509    4882 scope.go:122] "RemoveContainer" containerID="932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166"
Nov 01 04:33:53.745957 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:33:53.745928    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166\": container with ID starting with 932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166 not found: ID does not exist" containerID="932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166"
Nov 01 04:33:53.746046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.745969    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166"} err="failed to get container status \"932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166\": rpc error: code = NotFound desc = could not find container \"932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166\": container with ID starting with 932d961dde548829528c5b736c20350b5a20521b1c9cca0855501b7839783166 not found: ID does not exist"
Nov 01 04:33:53.746046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.745994    4882 scope.go:122] "RemoveContainer" containerID="77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80"
Nov 01 04:33:53.755385 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.755365    4882 scope.go:122] "RemoveContainer" containerID="77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80"
Nov 01 04:33:53.755666 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:33:53.755647    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80\": container with ID starting with 77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80 not found: ID does not exist" containerID="77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80"
Nov 01 04:33:53.755722 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.755673    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80"} err="failed to get container status \"77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80\": rpc error: code = NotFound desc = could not find container \"77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80\": container with ID starting with 77c692366b322c36d1ed004ff0f4bef95413baae441bd52a15ac4401789f1a80 not found: ID does not exist"
Nov 01 04:33:53.755722 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.755689    4882 scope.go:122] "RemoveContainer" containerID="b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196"
Nov 01 04:33:53.763890 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.763871    4882 scope.go:122] "RemoveContainer" containerID="b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196"
Nov 01 04:33:53.764209 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:33:53.764181    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196\": container with ID starting with b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196 not found: ID does not exist" containerID="b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196"
Nov 01 04:33:53.764254 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.764218    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196"} err="failed to get container status \"b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196\": rpc error: code = NotFound desc = could not find container \"b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196\": container with ID starting with b174d9066c22d79274adbdc11d0f39930a8121b1761ed7567f9c24a030cb3196 not found: ID does not exist"
Nov 01 04:33:53.884198 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.884113    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"]
Nov 01 04:33:53.888406 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.888382    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps"]
Nov 01 04:33:53.990097 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.990054    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"]
Nov 01 04:33:53.995702 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:53.995675    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-865066/simulator-w2-865066-kserve-54d996bff6-8crkc"]
Nov 01 04:33:54.015840 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:54.015789    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-865066/distinct-w2-865066-kserve-864548c4cb-jpmps" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:33:54.096898 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:54.096867    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"]
Nov 01 04:33:54.100710 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:54.100687    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-865066/premium-w2-865066-kserve-fd656859c-h8thg"]
Nov 01 04:33:55.598494 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:55.598459    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1301538d-17a0-4021-aeda-db3130e114b1" path="/var/lib/kubelet/pods/1301538d-17a0-4021-aeda-db3130e114b1/volumes"
Nov 01 04:33:55.598890 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:55.598877    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" path="/var/lib/kubelet/pods/c92a7a3f-7ff0-42fc-846a-cc386e951275/volumes"
Nov 01 04:33:55.599269 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:33:55.599257    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" path="/var/lib/kubelet/pods/f8104497-11eb-49d7-b2b5-328dda72bb22/volumes"
Nov 01 04:33:58.722040 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:33:58.721999    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:34:02.889137 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:02.889097    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"]
Nov 01 04:34:02.935050 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:02.935013    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-865066-hf67v"]
Nov 01 04:34:03.599878 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:03.599788    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b" path="/var/lib/kubelet/pods/3ea5b60b-704b-43d3-ba7b-a7d785e2ae1b/volumes"
Nov 01 04:34:10.029507 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:10.029470    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"]
Nov 01 04:34:10.029941 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:10.029694    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy" containerID="cri-o://589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b" gracePeriod=30
Nov 01 04:34:10.280297 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:10.280219    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 04:34:10.280297 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:10.280284    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 04:34:11.190332 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:11.190302    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f87c44d4-d554c_ecee51df-9d85-4121-bfa9-473fa0d54088/authorino/0.log"
Nov 01 04:34:11.548493 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:11.548410    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-g8ngp_41c7a6e2-4a4f-44cb-9c73-b41108c72e30/manager/0.log"
Nov 01 04:34:11.797842 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:11.797802    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gd4v5_06243225-e7c0-41c7-b49d-4bfa4b423d16/manager/0.log"
Nov 01 04:34:14.905284 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:14.905252    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7nrj5_83ed9b69-e27e-451b-8ab9-2fc60f6621f9/manager/0.log"
Nov 01 04:34:15.285749 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.285723    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 04:34:15.419642 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.419610    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istiod-ca-cert\") pod \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") "
Nov 01 04:34:15.419812 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.419675    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-credential-socket\") pod \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") "
Nov 01 04:34:15.419812 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.419731    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-envoy\") pod \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") "
Nov 01 04:34:15.419812 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.419767    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-token\" (UniqueName: \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-token\") pod \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") "
Nov 01 04:34:15.419812 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.419792    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-certs\") pod \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") "
Nov 01 04:34:15.420014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.419821    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-socket\") pod \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") "
Nov 01 04:34:15.420014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.419870    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-data\") pod \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") "
Nov 01 04:34:15.420014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.419901    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-podinfo\") pod \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") "
Nov 01 04:34:15.420014 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.419934    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-kube-api-access-2k2rl\" (UniqueName: \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-kube-api-access-2k2rl\") pod \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\" (UID: \"b34faf4a-79c3-4f04-928b-dce7aa9fed52\") "
Nov 01 04:34:15.420231 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.420032    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-credential-socket" pod "b34faf4a-79c3-4f04-928b-dce7aa9fed52" (UID: "b34faf4a-79c3-4f04-928b-dce7aa9fed52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:34:15.420231 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.420119    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-certs" pod "b34faf4a-79c3-4f04-928b-dce7aa9fed52" (UID: "b34faf4a-79c3-4f04-928b-dce7aa9fed52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:34:15.420231 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.420127    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istiod-ca-cert" pod "b34faf4a-79c3-4f04-928b-dce7aa9fed52" (UID: "b34faf4a-79c3-4f04-928b-dce7aa9fed52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:34:15.420231 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.420138    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-socket" pod "b34faf4a-79c3-4f04-928b-dce7aa9fed52" (UID: "b34faf4a-79c3-4f04-928b-dce7aa9fed52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:34:15.420454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.420320    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-data" pod "b34faf4a-79c3-4f04-928b-dce7aa9fed52" (UID: "b34faf4a-79c3-4f04-928b-dce7aa9fed52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:34:15.420454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.420345    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:34:15.420454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.420367    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istiod-ca-cert\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:34:15.420454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.420382    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-credential-socket\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:34:15.420454 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.420397    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-workload-certs\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:34:15.422006 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.421985    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-envoy" pod "b34faf4a-79c3-4f04-928b-dce7aa9fed52" (UID: "b34faf4a-79c3-4f04-928b-dce7aa9fed52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:34:15.422137 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.422005    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-token" pod "b34faf4a-79c3-4f04-928b-dce7aa9fed52" (UID: "b34faf4a-79c3-4f04-928b-dce7aa9fed52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:34:15.422137 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.422116    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-kube-api-access-2k2rl" pod "b34faf4a-79c3-4f04-928b-dce7aa9fed52" (UID: "b34faf4a-79c3-4f04-928b-dce7aa9fed52"). InnerVolumeSpecName "kube-api-access-2k2rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:34:15.422225 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.422144    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-podinfo" pod "b34faf4a-79c3-4f04-928b-dce7aa9fed52" (UID: "b34faf4a-79c3-4f04-928b-dce7aa9fed52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:34:15.521531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.521442    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-envoy\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:34:15.521531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.521471    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-token\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:34:15.521531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.521481    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-data\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:34:15.521531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.521490    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b34faf4a-79c3-4f04-928b-dce7aa9fed52-istio-podinfo\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:34:15.521531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.521501    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2rl\" (UniqueName: \"kubernetes.io/projected/b34faf4a-79c3-4f04-928b-dce7aa9fed52-kube-api-access-2k2rl\") on node \"ip-10-0-129-252.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 04:34:15.865327 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.865235    4882 generic.go:356] "Generic (PLEG): container finished" podID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerID="589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b" exitCode=0
Nov 01 04:34:15.865327 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.865276    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" event={"ID":"b34faf4a-79c3-4f04-928b-dce7aa9fed52","Type":"ContainerDied","Data":"589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b"}
Nov 01 04:34:15.865610 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.865348    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b" event={"ID":"b34faf4a-79c3-4f04-928b-dce7aa9fed52","Type":"ContainerDied","Data":"15f6eccfd21693928384a7d0e161a2bea6c8a20281916db194d348a3a0b14274"}
Nov 01 04:34:15.865610 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.865286    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"
Nov 01 04:34:15.865610 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.865379    4882 scope.go:122] "RemoveContainer" containerID="589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b"
Nov 01 04:34:15.874490 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.874473    4882 scope.go:122] "RemoveContainer" containerID="eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9"
Nov 01 04:34:15.882889 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.882873    4882 scope.go:122] "RemoveContainer" containerID="589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b"
Nov 01 04:34:15.883146 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:34:15.883124    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b\": container with ID starting with 589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b not found: ID does not exist" containerID="589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b"
Nov 01 04:34:15.883187 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.883159    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b"} err="failed to get container status \"589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b\": rpc error: code = NotFound desc = could not find container \"589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b\": container with ID starting with 589a5e42c6470d46b50cdcc90e69db815443a845f872ed675f175ad2eedf883b not found: ID does not exist"
Nov 01 04:34:15.883187 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.883179    4882 scope.go:122] "RemoveContainer" containerID="eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9"
Nov 01 04:34:15.883470 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:34:15.883452    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9\": container with ID starting with eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9 not found: ID does not exist" containerID="eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9"
Nov 01 04:34:15.883526 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:15.883478    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9"} err="failed to get container status \"eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9\": rpc error: code = NotFound desc = could not find container \"eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9\": container with ID starting with eadf2460f74210f6e49da38be22ffde94ef3cf88e4dbb00ead1cfaf62dc50fe9 not found: ID does not exist"
Nov 01 04:34:16.020459 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:16.020420    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"]
Nov 01 04:34:16.066188 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:16.066162    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-rzp2b"]
Nov 01 04:34:16.226046 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:16.226019    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lcsnr_20a09652-c72e-4f0f-a539-0dba5964c838/proxy/0.log"
Nov 01 04:34:17.221251 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:17.221216    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f87c44d4-d554c_ecee51df-9d85-4121-bfa9-473fa0d54088/authorino/0.log"
Nov 01 04:34:17.599811 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:17.599731    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" path="/var/lib/kubelet/pods/b34faf4a-79c3-4f04-928b-dce7aa9fed52/volumes"
Nov 01 04:34:27.778561 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:27.778523    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7nrj5_83ed9b69-e27e-451b-8ab9-2fc60f6621f9/manager/0.log"
Nov 01 04:34:28.726602 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:34:28.726565    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:34:29.131531 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:29.131421    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lcsnr_20a09652-c72e-4f0f-a539-0dba5964c838/proxy/0.log"
Nov 01 04:34:30.418480 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:30.418450    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0932abfc-xzj5f_a3b071e4-568c-40a8-9a9f-1faa2168aadb/revoke-keys/0.log"
Nov 01 04:34:30.627968 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:30.627938    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-87254a4d-hs89k_0978d086-0b2f-4a5f-a94e-6a67863a9966/revoke-keys/0.log"
Nov 01 04:34:31.053805 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:31.053773    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a0f0fd1b-lfhlk_4330d5aa-89dd-4a62-8373-3ca2500825be/revoke-keys/0.log"
Nov 01 04:34:31.284138 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:31.284110    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bff8b8d9-rh28n_e52816b2-9171-4f14-a409-e5316862c985/revoke-keys/0.log"
Nov 01 04:34:31.508496 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:31.508470    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d584266a-tvbpj_ac4811a7-193b-4797-bdcb-7a5f7e22ee63/revoke-keys/0.log"
Nov 01 04:34:31.728379 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:31.728349    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e9caabc2-rzb6k_d0039588-fe92-410b-ad5e-3cc921a48f44/revoke-keys/0.log"
Nov 01 04:34:31.954662 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:31.954627    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-4qgp8_24ec074e-6d28-431a-9327-25352b16dc0c/postgres/0.log"
Nov 01 04:34:33.419071 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:33.419040    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f87c44d4-d554c_ecee51df-9d85-4121-bfa9-473fa0d54088/authorino/0.log"
Nov 01 04:34:33.687742 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:33.687715    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-g8ngp_41c7a6e2-4a4f-44cb-9c73-b41108c72e30/manager/0.log"
Nov 01 04:34:33.907083 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:33.907050    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gd4v5_06243225-e7c0-41c7-b49d-4bfa4b423d16/manager/0.log"
Nov 01 04:34:36.090259 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:36.090224    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f85sv5q4_8da28ad3-267f-43d9-9c13-9981e3cce024/istio-proxy/0.log"
Nov 01 04:34:37.203283 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:37.203256    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh_8d06d61e-c4e6-465b-b8f1-8de9ab96a46b/istio-proxy/0.log"
Nov 01 04:34:37.840514 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:37.840482    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dcc875c8f-dlqlr_036c1722-9db5-406e-bc09-0aa6f215e1e0/router/0.log"
Nov 01 04:34:39.029945 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:39.029916    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-tzkq2_39634861-5963-4ce5-abaa-8aafd19668a7/main/0.log"
Nov 01 04:34:39.246387 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:39.246358    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-6756s_9499604f-575b-4971-b182-1073437989c7/main/0.log"
Nov 01 04:34:39.683819 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:39.683790    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tqv66_f26ff8f3-4414-4233-b7e3-343c78a767b7/main/0.log"
Nov 01 04:34:40.121165 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:40.121137    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-67x52_91fcd874-e74a-4bf5-a6ca-3842296d22b7/main/0.log"
Nov 01 04:34:49.664381 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664348    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jzwvp/must-gather-9trfd"]
Nov 01 04:34:49.664783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664744    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main"
Nov 01 04:34:49.664783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664756    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main"
Nov 01 04:34:49.664783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664772    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy"
Nov 01 04:34:49.664783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664778    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy"
Nov 01 04:34:49.664783 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664785    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerName="main"
Nov 01 04:34:49.664954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664791    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerName="main"
Nov 01 04:34:49.664954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664805    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main"
Nov 01 04:34:49.664954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664810    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main"
Nov 01 04:34:49.664954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664823    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0978d086-0b2f-4a5f-a94e-6a67863a9966" containerName="revoke-keys"
Nov 01 04:34:49.664954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664828    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0978d086-0b2f-4a5f-a94e-6a67863a9966" containerName="revoke-keys"
Nov 01 04:34:49.664954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664891    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy"
Nov 01 04:34:49.664954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664898    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1301538d-17a0-4021-aeda-db3130e114b1" containerName="main"
Nov 01 04:34:49.664954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664909    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92a7a3f-7ff0-42fc-846a-cc386e951275" containerName="main"
Nov 01 04:34:49.664954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664916    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8104497-11eb-49d7-b2b5-328dda72bb22" containerName="main"
Nov 01 04:34:49.664954 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664923    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0978d086-0b2f-4a5f-a94e-6a67863a9966" containerName="revoke-keys"
Nov 01 04:34:49.665256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.664996    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy"
Nov 01 04:34:49.665256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.665001    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy"
Nov 01 04:34:49.665256 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.665055    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34faf4a-79c3-4f04-928b-dce7aa9fed52" containerName="istio-proxy"
Nov 01 04:34:49.667914 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.667883    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jzwvp/must-gather-9trfd"
Nov 01 04:34:49.716626 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.716601    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jzwvp\"/\"default-dockercfg-s6jnz\""
Nov 01 04:34:49.722067 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.722042    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44e7cdea-d1c8-479e-96f0-8a027a4ce641-must-gather-output\") pod \"must-gather-9trfd\" (UID: \"44e7cdea-d1c8-479e-96f0-8a027a4ce641\") " pod="openshift-must-gather-jzwvp/must-gather-9trfd"
Nov 01 04:34:49.722187 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.722077    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92tdg\" (UniqueName: \"kubernetes.io/projected/44e7cdea-d1c8-479e-96f0-8a027a4ce641-kube-api-access-92tdg\") pod \"must-gather-9trfd\" (UID: \"44e7cdea-d1c8-479e-96f0-8a027a4ce641\") " pod="openshift-must-gather-jzwvp/must-gather-9trfd"
Nov 01 04:34:49.762606 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.762575    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jzwvp\"/\"openshift-service-ca.crt\""
Nov 01 04:34:49.762759 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.762720    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jzwvp\"/\"kube-root-ca.crt\""
Nov 01 04:34:49.764959 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.764940    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jzwvp/must-gather-9trfd"]
Nov 01 04:34:49.822667 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.822634    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44e7cdea-d1c8-479e-96f0-8a027a4ce641-must-gather-output\") pod \"must-gather-9trfd\" (UID: \"44e7cdea-d1c8-479e-96f0-8a027a4ce641\") " pod="openshift-must-gather-jzwvp/must-gather-9trfd"
Nov 01 04:34:49.822809 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.822673    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92tdg\" (UniqueName: \"kubernetes.io/projected/44e7cdea-d1c8-479e-96f0-8a027a4ce641-kube-api-access-92tdg\") pod \"must-gather-9trfd\" (UID: \"44e7cdea-d1c8-479e-96f0-8a027a4ce641\") " pod="openshift-must-gather-jzwvp/must-gather-9trfd"
Nov 01 04:34:49.822988 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.822969    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/44e7cdea-d1c8-479e-96f0-8a027a4ce641-must-gather-output\") pod \"must-gather-9trfd\" (UID: \"44e7cdea-d1c8-479e-96f0-8a027a4ce641\") " pod="openshift-must-gather-jzwvp/must-gather-9trfd"
Nov 01 04:34:49.880292 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.880262    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92tdg\" (UniqueName: \"kubernetes.io/projected/44e7cdea-d1c8-479e-96f0-8a027a4ce641-kube-api-access-92tdg\") pod \"must-gather-9trfd\" (UID: \"44e7cdea-d1c8-479e-96f0-8a027a4ce641\") " pod="openshift-must-gather-jzwvp/must-gather-9trfd"
Nov 01 04:34:49.982844 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:49.982813    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jzwvp/must-gather-9trfd"
Nov 01 04:34:50.347463 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:50.347439    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jzwvp/must-gather-9trfd"]
Nov 01 04:34:50.350296 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:34:50.350256    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44e7cdea_d1c8_479e_96f0_8a027a4ce641.slice/crio-3890cd56bdb5f2da3ee19d309fe942d46f0ecc469032adc80b50b4a922498a0d WatchSource:0}: Error finding container 3890cd56bdb5f2da3ee19d309fe942d46f0ecc469032adc80b50b4a922498a0d: Status 404 returned error can't find the container with id 3890cd56bdb5f2da3ee19d309fe942d46f0ecc469032adc80b50b4a922498a0d
Nov 01 04:34:50.352239 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:50.352225    4882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 04:34:51.061326 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:51.061291    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jzwvp/must-gather-9trfd" event={"ID":"44e7cdea-d1c8-479e-96f0-8a027a4ce641","Type":"ContainerStarted","Data":"3890cd56bdb5f2da3ee19d309fe942d46f0ecc469032adc80b50b4a922498a0d"}
Nov 01 04:34:58.106771 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:58.106730    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jzwvp/must-gather-9trfd" event={"ID":"44e7cdea-d1c8-479e-96f0-8a027a4ce641","Type":"ContainerStarted","Data":"e4d36424333b81310d8315541d638ae073db2d0dc44d47388894abfacea37255"}
Nov 01 04:34:58.107171 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:58.106779    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jzwvp/must-gather-9trfd" event={"ID":"44e7cdea-d1c8-479e-96f0-8a027a4ce641","Type":"ContainerStarted","Data":"2db9420773b46830065620d92ffac5900e25fae8425c41ea0d7e1fc01ea95421"}
Nov 01 04:34:58.214615 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:34:58.214549    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jzwvp/must-gather-9trfd" podStartSLOduration=2.53009175 podStartE2EDuration="9.21453434s" totalImagesPullingTime="6.68444259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:34:58.213798439 +0000 UTC m=+3166.775831456" watchObservedRunningTime="2026-11-01 04:34:58.21453434 +0000 UTC m=+3166.776567358"
Nov 01 04:34:58.738889 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:34:58.738849    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:35:01.828717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:01.828685    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7nrj5_83ed9b69-e27e-451b-8ab9-2fc60f6621f9/manager/0.log"
Nov 01 04:35:03.097300 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:03.097271    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zdw9t_b3f80953-074f-455a-9c88-6b3154926050/global-pull-secret-syncer/0.log"
Nov 01 04:35:03.195161 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:03.195132    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lcsnr_20a09652-c72e-4f0f-a539-0dba5964c838/proxy/0.log"
Nov 01 04:35:03.568625 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:03.568593    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6t6x6_6a4a6bb3-f059-4fcf-9d3f-af5667cf7f10/connection-checker/0.log"
Nov 01 04:35:03.967727 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:03.967696    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wgcvj_cd24318e-36ed-4fbb-9742-e6e2a9486283/connection-checker/0.log"
Nov 01 04:35:04.368322 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:04.368253    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-x52jq_cd1787d7-cd00-470d-b48f-c2d71e566dd2/connection-checker/0.log"
Nov 01 04:35:05.639170 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:05.639143    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mm8h5_ae127ade-3e32-4aa3-a528-324aa704bcdf/konnectivity-agent/0.log"
Nov 01 04:35:06.418844 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:06.418815    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-252.us-west-2.compute.internal_349f3d418ab214cf804b3c489f47ad32/haproxy/0.log"
Nov 01 04:35:06.632478 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:06.632446    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7nrj5_83ed9b69-e27e-451b-8ab9-2fc60f6621f9/manager/0.log"
Nov 01 04:35:07.876183 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:07.876153    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lcsnr_20a09652-c72e-4f0f-a539-0dba5964c838/proxy/0.log"
Nov 01 04:35:11.117982 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:11.117951    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7nrj5_83ed9b69-e27e-451b-8ab9-2fc60f6621f9/manager/0.log"
Nov 01 04:35:12.355892 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:12.355860    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lcsnr_20a09652-c72e-4f0f-a539-0dba5964c838/proxy/0.log"
Nov 01 04:35:12.606927 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:12.606848    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f87c44d4-d554c_ecee51df-9d85-4121-bfa9-473fa0d54088/authorino/0.log"
Nov 01 04:35:12.909754 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:12.909683    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-g8ngp_41c7a6e2-4a4f-44cb-9c73-b41108c72e30/manager/0.log"
Nov 01 04:35:12.986191 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:12.986161    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"]
Nov 01 04:35:12.992563 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:12.992543    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.081195 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.081167    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"]
Nov 01 04:35:13.128377 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.128348    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jth2x\"/\"openshift-service-ca.crt\""
Nov 01 04:35:13.145852 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.145826    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-proc\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.145978 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.145869    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-sys\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.146029 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.145978    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-podres\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.146088 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.146023    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d968\" (UniqueName: \"kubernetes.io/projected/f7795869-16b0-4b14-bb8d-ada59aeb3671-kube-api-access-8d968\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.146132 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.146103    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-lib-modules\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.146290 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.146270    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jth2x\"/\"default-dockercfg-j7x5g\""
Nov 01 04:35:13.146335 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.146298    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jth2x\"/\"kube-root-ca.crt\""
Nov 01 04:35:13.246921 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.246888    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-proc\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.247126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.246935    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-sys\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.247126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.246982    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-podres\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.247126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.247016    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-proc\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.247126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.247027    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d968\" (UniqueName: \"kubernetes.io/projected/f7795869-16b0-4b14-bb8d-ada59aeb3671-kube-api-access-8d968\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.247126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.247082    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-sys\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.247126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.247083    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-podres\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.247126 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.247089    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-lib-modules\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.247530 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.247140    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7795869-16b0-4b14-bb8d-ada59aeb3671-lib-modules\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.303793 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.303766    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gd4v5_06243225-e7c0-41c7-b49d-4bfa4b423d16/manager/0.log"
Nov 01 04:35:13.304066 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.304046    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d968\" (UniqueName: \"kubernetes.io/projected/f7795869-16b0-4b14-bb8d-ada59aeb3671-kube-api-access-8d968\") pod \"perf-node-gather-daemonset-5d52c\" (UID: \"f7795869-16b0-4b14-bb8d-ada59aeb3671\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.305867 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.305850    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:13.672189 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:13.672164    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"]
Nov 01 04:35:13.674865 ip-10-0-129-252 kubenswrapper[4882]: W1101 04:35:13.674833    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf7795869_16b0_4b14_bb8d_ada59aeb3671.slice/crio-216f074476b859be8f0d4cd23399a9296b89f766093ee919920106ed0baa92c8 WatchSource:0}: Error finding container 216f074476b859be8f0d4cd23399a9296b89f766093ee919920106ed0baa92c8: Status 404 returned error can't find the container with id 216f074476b859be8f0d4cd23399a9296b89f766093ee919920106ed0baa92c8
Nov 01 04:35:14.052653 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:14.052618    4882 scope.go:122] "RemoveContainer" containerID="9537da3989bdbd22714577f8783adc878cc29c2a9a1d9ce6f91ca6704772891e"
Nov 01 04:35:14.062012 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:14.061994    4882 scope.go:122] "RemoveContainer" containerID="091d5a18387b6a10a0c168f845b230d04f4cd0c0f143caf08bb84e2bf61801b9"
Nov 01 04:35:14.202541 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:14.202501    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c" event={"ID":"f7795869-16b0-4b14-bb8d-ada59aeb3671","Type":"ContainerStarted","Data":"8d75c05a6b4e5decfb7e37fb2bf5659b351e4d52c2afb33da41f49992671a97f"}
Nov 01 04:35:14.202717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:14.202592    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c" event={"ID":"f7795869-16b0-4b14-bb8d-ada59aeb3671","Type":"ContainerStarted","Data":"216f074476b859be8f0d4cd23399a9296b89f766093ee919920106ed0baa92c8"}
Nov 01 04:35:14.202717 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:14.202616    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:14.309622 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:14.309533    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c" podStartSLOduration=2.309521348 podStartE2EDuration="2.309521348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 04:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:35:14.309255498 +0000 UTC m=+3182.871288515" watchObservedRunningTime="2026-11-01 04:35:14.309521348 +0000 UTC m=+3182.871554364"
Nov 01 04:35:15.756489 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:15.756456    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f85sv5q4_8da28ad3-267f-43d9-9c13-9981e3cce024/istio-proxy/0.log"
Nov 01 04:35:20.219525 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:20.219496    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-5d52c"
Nov 01 04:35:20.608104 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:20.607994    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-pknwh_d50a0b39-914c-4b62-9b0b-4c6ab787f9d7/cluster-monitoring-operator/0.log"
Nov 01 04:35:21.140713 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:21.140676    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-599fb7fb95-qvpdh_8d06d61e-c4e6-465b-b8f1-8de9ab96a46b/istio-proxy/0.log"
Nov 01 04:35:21.780203 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:21.780171    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dcc875c8f-dlqlr_036c1722-9db5-406e-bc09-0aa6f215e1e0/router/0.log"
Nov 01 04:35:22.986492 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:22.986462    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2cmm_e3a6c29e-ea5b-498a-9c23-ad13dd2464a0/node-exporter/0.log"
Nov 01 04:35:23.380608 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:23.380532    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2cmm_e3a6c29e-ea5b-498a-9c23-ad13dd2464a0/kube-rbac-proxy/0.log"
Nov 01 04:35:23.773928 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:23.773904    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2cmm_e3a6c29e-ea5b-498a-9c23-ad13dd2464a0/init-textfile/0.log"
Nov 01 04:35:26.602037 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:26.602002    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f87c44d4-d554c_ecee51df-9d85-4121-bfa9-473fa0d54088/authorino/0.log"
Nov 01 04:35:26.716690 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:26.716661    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-g8ngp_41c7a6e2-4a4f-44cb-9c73-b41108c72e30/manager/0.log"
Nov 01 04:35:26.921983 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:26.921907    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gd4v5_06243225-e7c0-41c7-b49d-4bfa4b423d16/manager/0.log"
Nov 01 04:35:28.722741 ip-10-0-129-252 kubenswrapper[4882]: E1101 04:35:28.722698    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7rft8" containerName="extractor"
Nov 01 04:35:31.053785 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:31.053745    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"]
Nov 01 04:35:31.057060 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:31.057034    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"]
Nov 01 04:35:31.107629 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:31.107591    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f0fd1b-lfhlk"]
Nov 01 04:35:31.109962 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:31.109927    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9caabc2-rzb6k"]
Nov 01 04:35:31.429206 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:31.429108    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lbwxx_6d804e14-94cd-4c14-958f-66deb766ee5a/prometheus-operator-admission-webhook/0.log"
Nov 01 04:35:31.599627 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:31.599589    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4330d5aa-89dd-4a62-8373-3ca2500825be" path="/var/lib/kubelet/pods/4330d5aa-89dd-4a62-8373-3ca2500825be/volumes"
Nov 01 04:35:31.600219 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:31.600199    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0039588-fe92-410b-ad5e-3cc921a48f44" path="/var/lib/kubelet/pods/d0039588-fe92-410b-ad5e-3cc921a48f44/volumes"
Nov 01 04:35:36.035746 ip-10-0-129-252 kubenswrapper[4882]: I1101 04:35:36.035703    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-56c7694f85sv5q4_8da28ad3-267f-43d9-9c13-9981e3cce024/istio-proxy/0.log"